feat(cases): add test cases 9 and 10#26

Merged

constantinius merged 3 commits into

mainfrom

constantinius/feat/test-message-truncation

Jan 20, 2026

constantinius commented Jan 19, 2026

Collaborator

Added test cases 9-message-truncation and 10-binary-content-redaction for all JS and PY frameworks.

9-message-truncation: tests that messages get removed once the size limit has been reached and comparing against gen_ai.request.messages.original_length

10-binary-content-redaction: asserts that binary data is redacted

constantinius added 2 commits

January 19, 2026 10:17


          feat(cases): add test cases 9 and 10 for message truncation and blob …

105b50d

…redaction in Python


          feat(js): add tests for JS for cases 9 and 10

a6cd108

constantinius requested a review from obostjancic

January 19, 2026 09:22

github-actions Bot commented Jan 19, 2026 •

edited

Loading

🟡 AI SDK Integration Test Results

Status: 3 tests fixed, 34 still failing

Summary

Metric	main	PR	Change
Total Tests	28	56	+28
Passed	13	22	+9 ✅
Failed	15	34	+19 ⚠️

✅ Fixed

These tests were failing on main but are now passing:

js/langgraph :: 1-simple
js/langgraph :: 2-multi-step
py/google-genai :: 2-multi-step

🆕 New Tests

Passing (6):

✅ js/anthropic :: 9-message-truncation
✅ js/google-genai :: 9-message-truncation
✅ js/langchain :: 9-message-truncation
✅ js/langgraph :: 9-message-truncation
✅ js/openai :: 9-message-truncation
✅ js/vercel :: 9-message-truncation

Failing (22):

❌ js/anthropic :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.messages):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":[{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"[Filtered]\"}},{\"type\":\"text\",\"text\":\"What color is this image? Answer in one word.\"}]}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ js/google-genai :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.models):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":{\"inlineData\":{\"mimeType\":\"image/png\",\"data\":\"[Filtered]\"}}},{\"role\":\"user\",\"content\":{\"text\":\"What do you see in this image? Describe briefly.\"}}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ js/langchain :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":\"[{\\\"type\\\":\\\"text\\\",\\\"text\\\":\\\"What color is this image? Answer in one word.\\\"},{\\\"type\\\":\\\"image_url\\\",\\\"image_url\\\":{\\\"url\\\":\\\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==\\\"}}]\"}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ js/langgraph :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":\"[{\\\"type\\\":\\\"image_url\\\",\\\"image_url\\\":{\\\"url\\\":\\\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8z8BQz0AEYBxVSF+FABJADveWkH6oAAAAAElFTkSuQmCC\\\"}},{\\\"type\\\":\\\"text\\\",\\\"text\\\":\\\"What color is this image? Answer in one word.\\\"}]\"}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ js/openai :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What do you see in this image?\"},{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==\"}}]}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ js/vercel :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.generate_text):
       gen_ai.request.messages: mismatch (expected: {"type":"json_array","min_length":1,"contains":"[Blob substitute]"}, got: "[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Describe this image briefly.\"},{\"type\":\"file\",\"mediaType\":\"image/png\",\"data\":\"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==\"}]}]")
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:129:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/anthropic :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.operation.name: missing
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/anthropic :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.operation.name: missing
       gen_ai.request.messages: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/google-genai :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
       gen_ai.request.messages: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/google-genai :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/langchain :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/langchain :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg=="}}, {"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/langgraph :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/langgraph :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg=="}}, {"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/litellm :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/litellm :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg=="}}, {"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/openai :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/openai :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg=="}}, {"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/openai-agents :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/openai-agents :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant that analyzes images."}]}, {"role": "user", "content": [{"type": "text", "text": [{"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg==", "detail": "auto"}, {"type": "input_text", "text": "What color is this image? Answer in one word."}]}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/pydantic-ai :: 9-message-truncation

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

❌ py/pydantic-ai :: 10-binary-content-redaction

Error: Fixture validation failed:

Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.usage.input_tokens: missing
       gen_ai.usage.output_tokens: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant that analyzes images."}]}, {"role": "user", "content": [{"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/testing-ai-sdk-integrations/testing-ai-sdk-integrations/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

Test Matrix

SDK	1-simple	10-binary-content-redaction	2-multi-step	9-message-truncation
js/anthropic	✅	❌🆕	✅	✅🆕
js/google-genai	✅	❌🆕	✅	✅🆕
js/langchain	✅	❌🆕	✅	✅🆕
js/langgraph	✅🔧	❌🆕	✅🔧	✅🆕
js/openai	✅	❌🆕	✅	✅🆕
js/vercel	❌	❌🆕	❌	✅🆕
py/anthropic	❌	❌🆕	❌	❌🆕
py/google-genai	✅	❌🆕	✅🔧	❌🆕
py/langchain	❌	❌🆕	❌	❌🆕
py/langgraph	❌	❌🆕	❌	❌🆕
py/litellm	✅	❌🆕	✅	❌🆕
py/openai	✅	❌🆕	✅	❌🆕
py/openai-agents	❌	❌🆕	❌	❌🆕
py/pydantic-ai	❌	❌🆕	❌	❌🆕

Legend: ✅ Pass | ❌ Fail | ✅🔧 Fixed | ❌📉 Regressed | ✅🆕 New (pass) | ❌🆕 New (fail) | 🗑️ Removed

Generated by AI SDK Integration Tests

obostjancic approved these changes

View reviewed changes


          fix: using a static image instad of creating one all the time

b76d9cf

constantinius merged commit 2c0212b into main

7 checks passed

constantinius deleted the constantinius/feat/test-message-truncation branch

January 20, 2026 10:52

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet