TestProvider for Unit Testing

[willbakst]

Description

Note

This is a rough proposal. It will be easier to clarify some details during implementation.

Problem

Users want to unit test Mirascope applications without making LLM calls. The current approach isn't documented or ergonomic, and we're missing key native functionality to make this a quality experience.

Solution

Add llm.TestProvider and an llm.test_provider context manager that intercepts all LLM calls at the provider level.

Why provider-level? The model.provider property is dynamic and looks up from the registry on every call. This means a registered TestProvider intercepts all calls, even from direct llm.Model() instantiation.

Proposed API

Basic Usage

from mirascope import llm

# Handles setting the registry and cleaning it up so it's scoped to this context
with llm.test_provider("openai/", responses=["Hello!"]) as provider:
    model = llm.Model("openai/gpt-4o")
    response = model.call("Hi")

    assert response.content == "Hello!"
    assert provider.call_count == 1

Agent Testing

from mirascope import llm

@llm.tool
def get_weather(self) -> str:
        return f"72°F and sunny in {self.location}"

@llm.agent("openai/gpt-4o", tools=[get_weather])
def weather_agent(question: str) -> str:
    return question

# Mock the full agent loop: tool call → final response
with llm.test_provider("openai/", responses=[
    llm.TestResponse(tool_calls=[get_weather(location="San Francisco")]),
    llm.TestResponse(content="Based on the current conditions, it's 72°F and sunny in San Francisco. Great weather for a walk!"),
]):
    result = weather_agent("What's the weather in SF?")
    assert "72°F" in result
    assert "sunny" in result

Streaming

String responses auto-work for streaming:

with llm.test_provider("openai/", responses=["Hello!"]):
    for chunk in model.stream("Hi"):
        print(chunk.content)  # Streams character by character

# Or explicit chunks for fine-grained control
with llm.test_provider("openai/", responses=[
    llm.TestResponse(chunks=["Hello", " world", "!"]),
]):
    chunks = [c.content for c in model.stream("Hi")]
    assert chunks == ["Hello", " world", "!"]

Response Factory

with llm.test_provider("openai/", response_factory=lambda messages, **_: f"You said: {messages[-1].content}"):
    assert model.call("test").content == "You said: test"

Scope

• llm.TestProvider - Provider returning mock responses
• llm.test_provider - Context manager with automatic cleanup
• llm.TestResponse - Full response mocking (content, tool calls, chunks)
• Auto-streaming from string/content responses
• Inspection: call_count, calls, last_messages