Skip to content

METHODOLOGY.md

Latest commit

 

History

History
257 lines (200 loc) · 9.07 KB

METHODOLOGY.md

File metadata and controls

257 lines (200 loc) · 9.07 KB
Raw

Benchmark Methodology

Overview

This benchmark suite compares LLM proxy gateways using VidaiMock as a consistent backend. All gateways route to identical backend conditions, isolating proxy overhead from LLM latency.

Gateways Tested:

  • VidaiServer L1/L2/L3 (Rust)
  • Bifrost (Go)
  • LiteLLM (Python)
  • Portkey (Node.js)

Principles:

  • Same backend (VidaiMock)
  • Same hardware
  • Same load patterns
  • Default configurations
  • Clean state between tests

Architecture

k6 (Load Generator)
    │
    ├─── Baseline ────────────────────┐
    │                                 │
    ├─── VidaiServer L1/L2/L3 ───┐    │
    │                            │    │
    ├─── LiteLLM ────────────────┼────┼──► VidaiMock (8100)
    │                            │    │
    ├─── Bifrost ────────────────┤    │
    │                            │    │
    └─── Portkey ────────────────┘    │
                                      │
    Ports:                            │
      VidaiServer: 3000               │
      LiteLLM:     4000               │
      Bifrost:     8080               │
      Portkey:     8787               │
                                      ▼
                              VidaiMock Template
                              (Dynamic Responses)

VidaiMock Per-Request Control

Unlike traditional mock servers that require profile switching and restarts, VidaiMock accepts per-request configuration via HTTP headers:

Header Values Effect
X-Response-Size small, medium, large Controls response size (~1KB, ~11KB, ~42KB)
X-Vidai-Latency Integer (ms) Injects artificial latency
X-Vidai-Jitter Float (0-1) Adds random jitter to latency
X-Vidai-Chaos-Drop Integer (0-100) Percentage of requests to fail

Template Example (Tera/Jinja2):

{%- set size = headers['x-response-size'] | default(value='small') -%}
{%- if size == 'large' -%}
  {%- set repeat_count = 320 -%}
{%- elif size == 'medium' -%}
  {%- set repeat_count = 80 -%}
{%- else -%}
  {%- set repeat_count = 5 -%}
{%- endif -%}

This enables:

  • Mixed payload distributions within a single test
  • Per-request latency simulation
  • Chaos injection without service restart
  • Full k6 control over test parameters

Test Dimensions

1. Sanity Test

Quick validation of all dimensions before full benchmarks.

Check What Expected
Basic request Simple completion 200 OK
Response sizes Small/medium/large Correct byte ranges
Latency injection 200ms header ~200ms response time
Chaos injection 50% error rate Mix of 200/500
Streaming SSE format data: prefixed chunks
Tool calling Function call response tool_calls in response

Duration: ~5 minutes per gateway

2. Throughput Test (Stepwise)

Measures latency and error rate at increasing RPS levels.

RPS Target Duration per Step Purpose
200-500 15s each Warmup / baseline
1000-3000 15s each Light load
4000-6000 15s each Medium load
8000-12000 15s each High load / breaking point

RPS Steps: 200, 500, 1000, 1500, 2000, 3000, 4000, 5000, 6000, 8000, 10000, 12000

VU Allocation (single-machine optimized):

// Each VU achieves ~25-35 RPS with instant responses
// More aggressive allocation to reach 12K RPS
preAllocatedVUs = min(max(rps / 25, 10), 600)
maxVUs = min(max(rps / 20, 20), 800)

Key Metrics:

  • p50, p95, p99 latency at each step
  • Error rate at each step
  • Max stable RPS (last step with <5% errors)

3. Chaos / Resilience Test

Measures how gateways handle backend failures.

Configuration:

  • Error injection: 10% (via X-Vidai-Chaos-Drop: 10)
  • Duration: 60s
  • Target RPS: 500

Error Classification:

  • Expected errors: HTTP 500 with "Simulated" in body (chaos injection working correctly)
  • Unexpected errors: Any other failure (gateway problem)

Resilience Grading:

Grade Criteria
A+ Absorbs errors (retry/circuit breaker)
A Clean passthrough (amplification ≤ 1.0)
B-C Minor amplification (1.05-1.1x)
D-F Major amplification or unexpected errors

VidaiServer Layers

Layer Config Features Purpose
L1 config.layer1-base.yaml Routing, SQLite Pure proxy overhead
L2 config.layer2-guardrails.yaml + Guardrails Security feature cost
L3 config.layer3-enterprise.yaml + PostgreSQL Full enterprise overhead

Fairness Measures

  1. Same Backend: All gateways route to identical VidaiMock instance
  2. Same Hardware: All tests on same machine, same run
  3. Same Load Pattern: Identical k6 scenarios for each gateway
  4. Default Configs: No special tuning beyond required setup
  5. Clean State: Database reset between tests
  6. Connection Reuse: k6 configured with connection pooling
  7. Warm-up: 5-10s ramp-up before measurement

Gateway-Specific Notes

Header Forwarding Support

VidaiMock uses HTTP headers for per-request control (response size, latency, chaos). Each gateway handles custom header forwarding differently:

Gateway Header Forwarding Notes
VidaiServer ✅ Full support Forwards all headers by default
LiteLLM ✅ Full support Requires forward_client_headers_to_llm_api: true
Bifrost ✅ Full support Uses x-bf-eh-* prefix (e.g., x-bf-eh-X-Response-Size)
Portkey ⚠️ Limited Self-hosted gateway does not support x-portkey-forward-headers

Portkey Limitation

The self-hosted open-source Portkey gateway does not forward custom headers to the backend, even with the documented x-portkey-forward-headers header. This is a limitation of the open-source version; the cloud version (api.portkey.ai) may support this feature.

Impact on Benchmarks:

  • ✅ Throughput tests: Valid (uses default VidaiMock responses)
  • ✅ Streaming tests: Valid (no headers needed)
  • ✅ Tool calling tests: Valid (no headers needed)
  • ❌ Response size tests: Cannot control response size
  • ❌ Latency simulation tests: Cannot inject artificial latency
  • ❌ Chaos/resilience tests: Cannot inject errors

Portkey results for payload size, latency, and chaos dimensions should be interpreted with this limitation in mind or excluded from those comparisons.

Known Limitations

  1. Single Machine: No network latency simulation; all services share CPU/memory
  2. Synthetic Backend: VidaiMock responses are instant (unless latency injected)
  3. Point-in-Time: Results depend on specific software versions
  4. VU Contention: At very high RPS, k6 itself may become a bottleneck
  5. Portkey Headers: Self-hosted Portkey cannot forward custom headers (see above)

Metrics Collected

Metric Source Description
p50, p95, p99 latency k6 Percentile latencies
Error rate k6 Failed requests / total requests
Achieved RPS k6 Actual requests/second delivered
Max stable RPS Analysis Last RPS step with <5% errors
Error amplification Analysis Gateway errors / backend errors
Resilience grade Analysis A+ to F based on error handling

Output Structure

results/<timestamp>/
├── sanity_baseline.json
├── sanity_vidai_layer1.json
├── throughput_baseline.json
├── throughput_vidai_layer1.json
├── chaos_baseline.json
├── charts/
│   ├── stepwise_analysis.png
│   ├── throughput_comparison.png
│   ├── chaos_resilience.png
│   ├── node_count_comparison.png
│   └── SUMMARY.md
└── *.log (raw k6 output)

Reproduce

# Setup (one-time)
./scripts/setup.sh
source configs/env/enterprise.env && source configs/env/admin.env

# Start services
./scripts/start_services.sh mac competitors

# Validate VidaiMock
./scripts/validate_vidaimock.sh all

# Run benchmarks
./run_benchmark.sh sanity      # Quick validation (~5 min)
./run_benchmark.sh throughput  # Full throughput (~60 min)
./run_benchmark.sh chaos       # Resilience (~30 min)
./run_benchmark.sh all         # Complete suite (~2 hrs)

Interpreting Results

Throughput Charts

  • p95 Latency vs Load: Lower is better. Watch for exponential growth.
  • Achieved vs Target RPS: Diagonal = keeping up. Plateau = saturated.
  • Error Rate vs Load: 5% is our threshold. Beyond = unstable.

Chaos Charts

  • Error Classification: Stacked bar. Orange = expected (good). Red = unexpected (bad).
  • Resilience Grade: A+ means the gateway helps. F means it hurts.
  • Error Amplification: 1.0x = passthrough. >1.0x = gateway adds failures.

Node Count Chart

Infrastructure economics: how many nodes needed for 10K RPS? Lower is better (fewer servers = lower cost).