Skip to content

VIDAIMOCK_API_REFERENCE.md

Latest commit

 

History

History
268 lines (217 loc) · 7.06 KB

VIDAIMOCK_API_REFERENCE.md

File metadata and controls

268 lines (217 loc) · 7.06 KB
Raw

VidaiMock API Reference for Benchmarking

This document lists all VidaiMock capabilities used by the benchmark suite.

Base URL

http://localhost:8100

Health Check

curl -sf http://localhost:8100/health
# Response: {"status":"ok"}

Chat Completions Endpoint

Basic Request

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}]}'

Response Size Control

Control response payload size via X-Response-Size header.

Small Response (~1KB, 5 repetitions)

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Response-Size: small" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

Medium Response (~11KB, 80 repetitions)

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Response-Size: medium" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

Large Response (~42KB, 320 repetitions)

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Response-Size: large" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

Latency Injection

Add artificial latency via X-Vidai-Latency header (milliseconds).

200ms Latency

time curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Vidai-Latency: 200" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'
# Expected: ~200ms response time

500ms Latency (Simulating Slow LLM)

time curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Vidai-Latency: 500" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'
# Expected: ~500ms response time

Latency with Jitter

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Vidai-Latency: 400" \
  -H "X-Vidai-Jitter: 0.2" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'
# Expected: 320-480ms (400ms ± 20%)

Chaos/Error Injection

Inject errors via X-Vidai-Chaos-Drop header (percentage 0-100).

100% Error Rate (Always Fail)

curl -s -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Vidai-Chaos-Drop: 100" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}' \
  -w "\nHTTP Status: %{http_code}\n"
# Response: HTTP 500, Body: "Simulated Internal Server Error"

50% Error Rate

# Run multiple times to see mix of success/failure
for i in {1..10}; do
  curl -sf -o /dev/null -w "%{http_code}\n" -X POST http://localhost:8100/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "X-Vidai-Chaos-Drop: 50" \
    -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'
done
# Expected: Mix of 200 and 500 status codes

Other Chaos Headers

# Malformed JSON response
curl -X POST http://localhost:8100/v1/chat/completions \
  -H "X-Vidai-Chaos-Malformed: 100" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

# Slow trickle (partial response)
curl -X POST http://localhost:8100/v1/chat/completions \
  -H "X-Vidai-Chaos-Trickle: 100" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

# Disconnect mid-stream
curl -X POST http://localhost:8100/v1/chat/completions \
  -H "X-Vidai-Chaos-Disconnect: 100" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

Streaming Responses

Request streaming via stream: true in request body.

Basic Streaming

curl -sN -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}],"stream":true}'
# Response: SSE format with data: prefix
# data: {"id":"chatcmpl-mock","object":"chat.completion.chunk",...}

Tool/Function Calling

Include tools array in request to get tool_calls response.

Tool Calling Request

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}
      }
    }]
  }'

Expected Response

{
  "id": "chatcmpl-...",
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1234",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"location\": \"Mock City\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": {"prompt_tokens": 15, "completion_tokens": 25, "total_tokens": 40}
}

Combined Headers

Headers can be combined for complex scenarios.

Large Response with Latency

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Response-Size: large" \
  -H "X-Vidai-Latency: 300" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

Small Response with Chaos

curl -sf -X POST http://localhost:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Response-Size: small" \
  -H "X-Vidai-Chaos-Drop: 10" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"test"}]}'

k6 Error Detection

When using chaos injection, k6 must distinguish expected vs unexpected errors:

Condition Expected? How to Detect
500 + "Simulated" body + chaos header set Yes Body contains "Simulated"
500 without chaos header No Actual proxy/upstream error
500 + different error message No Actual error

k6 Check Example

// In k6, check if error was expected (chaos) or unexpected
const chaosRate = parseInt(__ENV.CHAOS_RATE || '0');
const isChaosEnabled = chaosRate > 0;

const res = http.post(url, payload, { headers });

if (res.status === 500 && isChaosEnabled && res.body.includes('Simulated')) {
  // Expected chaos error - count separately
  expectedErrors.add(1);
} else if (res.status !== 200) {
  // Unexpected error - this is a real problem
  unexpectedErrors.add(1);
}

Validation Script

Run the validation script to verify VidaiMock is configured correctly:

./scripts/validate_vidaimock.sh all

Individual tests:

./scripts/validate_vidaimock.sh response-sizes
./scripts/validate_vidaimock.sh latency
./scripts/validate_vidaimock.sh chaos
./scripts/validate_vidaimock.sh streaming
./scripts/validate_vidaimock.sh tools
./scripts/validate_vidaimock.sh tokens