Audit Phase 4: Tracing Middleware

[nycomp]

Phase 4: Tracing Middleware

Parent: #424

Tasks

• Create internal audit client (campus.audit.client)
• Create tracing core logic (start_span, end_span, build_span_from_context)
• Create ingestion pipeline (async via ThreadPoolExecutor)
• Register middleware hooks (deploy.py)
• Start span, generate/reuse trace_id (uid.generate_trace_id exists)
• Complete span, record duration, ingest
• 32-char hex (OpenTelemetry-compatible) - uid.generate_trace_id()
• 16-char hex - uid.generate_span_id()
• Safe JSON extraction
• Extract with 64KB truncation
• Strip Authorization header before storage
• Echo X-Request-ID header in response
• Async span ingestion to avoid blocking requests
• Handle edge cases (streaming responses, errors, etc.)

Acceptance Criteria

• Middleware captures all requests automatically
• Trace IDs are consistent across parent-child spans
• Authorization headers are stripped
• Response bodies truncated at 64KB
• Integration test proves spans are recorded
• No performance impact on normal requests (< 1ms overhead)

References

• PRD §4.3: Design Decisions (Authorization stripping)
• PRD §9.1: Middleware-Based Ingestion
• PRD §9.3: Sensitive Data
• PRD §9.4: Body Truncation
• Plan §6: Middleware for Trace Capture

---

Status: Implementation complete (~90%), pending integration tests and performance benchmarking

Last Updated: April 10, 2026 - Created internal audit client and implemented full tracing middleware

[nycomp]

Implementation Status Update (April 10, 2026)

✅ What's Implemented:

• ID Generation: 32-char hex trace IDs and 16-char hex span IDs via uid module (OpenTelemetry-compatible)
• Data Model: Complete TraceSpan model with all required fields
• Database Schema: Migration 003 with proper indexes
• API Resources: Full CRUD operations for traces in campus/audit/resources/traces.py
• Middleware Structure: Hooks registered in deploy.py for auth/api deployments

❌ What's Missing (Core Functions Not Implemented):

The campus/audit/middleware/tracing.py file has only TODO stubs:

• start_span() - not implemented
• end_span() - not implemented
• build_span_from_context() - not implemented
• ingest_span() - not implemented

🚧 Outstanding Work:

• Request/response body capture with 64KB truncation
• Authorization header stripping before storage
• X-Request-ID header generation/echoing
• Async span ingestion to avoid blocking
• Edge case handling (streaming responses, errors)
• Integration with auth middleware for client_id/user_id
• Integration tests proving spans are recorded

📊 Progress Estimate: ~30% complete

The foundation is solid (models, API, IDs) but the actual request capturing logic is entirely missing.

[nycomp]

Implementation Update (April 10, 2026)

✅ New Implementation Complete:

Created internal audit client (campus.audit.client):

• Follows campus-api-python structure (ResourceRoot, Resource pattern)
• Uses campus.common.http.DefaultClient for HTTP
• Supports all trace operations: ingest, ingest_batch, list, get_tree
• Automatic environment-based URL resolution (development/testing/staging/production)

Implemented tracing middleware (campus.audit.middleware.tracing):

• ✅ start_span(): Generates/reuses trace_id, stores timing in flask.g
• ✅ end_span(): Completes span, calculates duration, echoes X-Request-ID
• ✅ build_span_from_context(): Extracts request/response data
• ✅ _extract_request_body(): JSON/form/text with safety checks
• ✅ _extract_response_body(): 64KB truncation, handles streaming
• ✅ Async ingestion via ThreadPoolExecutor (non-blocking)
• ✅ Authorization header stripping
• ✅ client_id/user_id from flask.g (populated by auth middleware)

📋 Remaining Items:

Testing & Integration:

• Unit tests for client (campus.audit.client)
• Unit tests for middleware functions
• Integration test proving spans are recorded end-to-end
• Performance benchmark (verify < 1ms overhead)
• Edge case handling (errors, timeouts, audit service unavailable)

Documentation:

• Update issue Audit Phase 4: Tracing Middleware #428 checklist
• Document environment variables (AUDIT_API_URL, AUDIT_API_KEY)

📁 Files Created/Modified:

• campus/audit/client/__init__.py - AuditClient entry point
• campus/audit/client/v1/__init__.py - v1 API module
• campus/audit/client/v1/root.py - AuditRoot resource
• campus/audit/client/v1/traces.py - Traces, Trace, Span resources
• campus/audit/client/interface.py - Resource base classes
• campus/audit/middleware/tracing.py - Full implementation (was TODOs)

[nycomp]

Refactor Complete (April 10, 2026)

✅ Client Refactored to Match Campus API Schema

Changes made to campus.audit.client:

1. Standard verbs - Changed to match campus-api-python:

   • ingest() → new() (standard create verb)
   • ingest_batch() → new(*spans) (varargs for batch)

2. Proper base classes:

   • Traces now extends ResourceCollection (was ResourceRoot)
   • Paths have trailing slashes (path = "traces/")

3. Nested classes following campus-api-python pattern:

   • Traces.Trace - Single trace resource
   • Traces.Spans - Spans collection (nested)
   • Traces.Spans.Span - Single span resource

4. Bracket access for IDs:

   • client.traces[trace_id] - Get trace by ID
   • client.traces[trace_id].spans[span_id] - Get span by ID

5. Property access for named sub-resources:

   • client.traces[trace_id].spans - Spans sub-resource (property)

New API usage pattern:

from campus.audit.client import AuditClient

client = AuditClient()

# Ingest spans (uses standard 'new' verb)
client.traces.new(span_data)           # Single
client.traces.new(s1, s2, s3)          # Batch

# List traces
client.traces.list(limit=10)

# Search traces
client.traces.search(path="/api/v1/users")

# Get trace tree (bracket for ID, property for sub-resource)
tree = client.traces[trace_id].get_tree()

# Get span (nested bracket access)
span = client.traces[trace_id].spans[span_id].get()

Middleware updated:

• _ingest_span_async() now calls client.traces.new(span)

All Campus API schema conventions are now followed.