Release 1.1.5 (#2202)

# Changelog

## New Features:

- **Audio Responses:** Agents can now deliver audio responses (both with
streaming and non-streaming).
    - The audio is in the `agent.run_response.response_audio`.
- This only works with `OpenAIChat` with the `gpt-4o-audio-preview`
model. See [their docs](https://platform.openai.com/docs/guides/audio)
for more on how it works. For example
        
        ```python
        from agno.agent import Agent
        from agno.models.openai import OpenAIChat
        from agno.utils.audio import write_audio_to_file
        
        agent = Agent(
            model=OpenAIChat(
                id="gpt-4o-audio-preview",
modalities=["text", "audio"], # Both text and audio responses are
provided.
                audio={"voice": "alloy", "format": "wav"},
            ),
        )
        agent.print_response(
            "Tell me a 5 second story"
        )
        if agent.run_response.response_audio is not None:
            write_audio_to_file(
audio=agent.run_response.response_audio.base64_audio,
filename=str(filename)
            )
        ```
        
- See the [audio_conversation_agent
cookbook](https://github.com/agno-agi/agno/blob/main/cookbook/playground/audio_conversation_agent.py)
to test it out on the Agent Playground.
- **Image understanding support for [Together.ai](http://together.ai/)
and XAi**: You can now give images to agents using models from XAi and
[Together.ai](http://together.ai/).
- **Webex Tool**: Added a tool for sending messages on
[Webex](https://www.webex.com/).
- **Upstash Vector DB:** Added support for
[Upstash](https://upstash.com/).

## Improvements:

- **Automated Tests:** Added integration tests for all models. Most of
these will be run on each pull request, with a suite of integration
tests run before a new release is published.
- **Grounding and Search with Gemini:** [Grounding and
Search](https://ai.google.dev/gemini-api/docs/grounding?lang=python) can
be used to improve the accuracy and recency of responses from the Gemini
models.

## Bug Fixes:

- **Structured output updates**: Fixed various cases where native
structured output was not used on models
- **Ollama tool parsing**: Fixed cases for Ollama with tools with
optional parameters
- **Gemini Memory Summariser**: Fixed cases where Gemini models were
used as the memory summariser.
- **Gemini Auto Tool Calling:** Enabled automatic tool calling when
tools are provided, aligning behavior with other models.
- **FixedSizeChunking issue with overlap:** Fixed issue where chunking
would fail if overlap was set.
- **Claude Tools with Multiple Types:** Fixed an issue where Claude
tools would break when handling a union of types in parameters.
- **JSON Response Parsing:** Fixed cases where JSON model responses
returned quoted strings within dictionary values.

Author: dirkbrnd

.github/workflows/main_test.yml

@@ -3,10 +3,8 @@ name: Main Validation
 on:
   push:
     branches:
-      - 'main'  # Run on all branches
-  pull_request:
-    branches:
-      - 'release*'  # Run on PRs targeting release branches
+      - 'main'  # Run on main
+      - '*release*'  # Run on release
 
 jobs:
   style-check:
@@ -53,6 +51,7 @@ jobs:
       fail-fast: true  # Stop all matrix jobs if one fails
     env:
       OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -86,6 +85,8 @@ jobs:
       fail-fast: true  # Stop all matrix jobs if one fails
     env:
       GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+      GOOGLE_CLOUD_PROJECT: ${{ secrets.GOOGLE_CLOUD_PROJECT }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -119,6 +120,7 @@ jobs:
       fail-fast: true  # Stop all matrix jobs if one fails
     env:
       ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -151,8 +153,10 @@ jobs:
         python-version: ["3.12"]
       fail-fast: true
     env:
-      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
-      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      AWS_ACCESS_KEY_ID: ${{ secrets.BEDROCK_AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.BEDROCK_AWS_SECRET_ACCESS_KEY }}
+      AWS_REGION: "us-east-1"
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -186,6 +190,7 @@ jobs:
       fail-fast: true
     env:
       CO_API_KEY: ${{ secrets.CO_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -252,6 +257,7 @@ jobs:
       fail-fast: true
     env:
       FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -285,6 +291,7 @@ jobs:
       fail-fast: true
     env:
       GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -318,6 +325,7 @@ jobs:
       fail-fast: true
     env:
       MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -344,6 +352,7 @@ jobs:
 
   # Run tests for Nvidia
   test-nvidia:
+    if: false # Disable nvidia tests until we get a test account
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -374,16 +383,17 @@ jobs:
         run: |
           source .venv/bin/activate
           python -m pytest ./libs/agno/tests/integration/models/nvidia
-
   # Run tests for OpenRouter
   test-openrouter:
+    if: false # Disable openrouter tests until we get a test account
     runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.12"]
       fail-fast: true
     env:
       OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -476,13 +486,15 @@ jobs:
 
   # Run tests for Together
   test-together:
+    if: false # The tests take too long to run
     runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.12"]
       fail-fast: true
     env:
       TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -516,6 +528,7 @@ jobs:
       fail-fast: true
     env:
       XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
+      EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}

cookbook/agent_concepts/multimodal/audio_streaming.py

@@ -0,0 +1,48 @@
+import base64
+import wave
+from pathlib import Path
+from typing import Iterator
+
+from agno.agent import Agent, RunResponse  # noqa
+from agno.models.openai import OpenAIChat
+
+# Audio Configuration
+SAMPLE_RATE = 24000  # Hz (24kHz)
+CHANNELS = 1  # Mono (Change to 2 if Stereo)
+SAMPLE_WIDTH = 2  # Bytes (16 bits)
+
+# Provide the agent with the audio file and audio configuration and get result as text + audio
+agent = Agent(
+    model=OpenAIChat(
+        id="gpt-4o-audio-preview",
+        modalities=["text", "audio"],
+        audio={
+            "voice": "alloy",
+            "format": "pcm16",
+        },  # Only pcm16 is supported with streaming
+    ),
+)
+output_stream: Iterator[RunResponse] = agent.run(
+    "Tell me a 10 second story", stream=True
+)
+
+filename = "tmp/response_stream.wav"
+
+# Open the file once in append-binary mode
+with wave.open(str(filename), "wb") as wav_file:
+    wav_file.setnchannels(CHANNELS)
+    wav_file.setsampwidth(SAMPLE_WIDTH)
+    wav_file.setframerate(SAMPLE_RATE)
+
+    # Iterate over generated audio
+    for response in output_stream:
+        if response.response_audio:
+            if response.response_audio.transcript:
+                print(response.response_audio.transcript, end="", flush=True)
+            if response.response_audio.content:
+                try:
+                    pcm_bytes = base64.b64decode(response.response_audio.content)
+                    wav_file.writeframes(pcm_bytes)
+                except Exception as e:
+                    print(f"Error decoding audio: {e}")
+print()

libs/agno/agno/agent/agent.py

@@ -2433,9 +2433,7 @@ def _transfer_task_to_agent(
             True
             if (
                 member_agent.response_model is not None
-                and member_agent.structured_outputs
                 and member_agent.model is not None
-                and member_agent.model.supports_structured_outputs
             )
             else False
         )

libs/agno/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agno"
-version = "1.1.4"
+version = "1.1.5"
 description = "Agno: a lightweight framework for building multi-modal Agents"
 requires-python = ">=3.7,<4"
 readme = "README.md"

libs/agno/tests/integration/__init__.py

@@ -3,7 +3,7 @@
 
 from agno.agent import Agent, RunResponse  # noqa
 from agno.models.anthropic import Claude
-from agno.storage.agent.postgres import PostgresAgentStorage
+from agno.storage.agent.sqlite import SqliteAgentStorage
 
 
 def _assert_metrics(response: RunResponse):
@@ -95,17 +95,7 @@ def test_with_memory():
     assert [m.role for m in agent.memory.messages] == ["system", "user", "assistant", "user", "assistant"]
 
     # Test metrics structure and types
-    input_tokens = response2.metrics["input_tokens"]
-    output_tokens = response2.metrics["output_tokens"]
-    total_tokens = response2.metrics["total_tokens"]
-
-    assert isinstance(input_tokens[0], int)
-    assert input_tokens[0] > 0
-    assert isinstance(output_tokens[0], int)
-    assert output_tokens[0] > 0
-    assert isinstance(total_tokens[0], int)
-    assert total_tokens[0] > 0
-    assert total_tokens[0] == input_tokens[0] + output_tokens[0]
+    _assert_metrics(response2)
 
 
 def test_structured_output():
@@ -128,10 +118,9 @@ class MovieScript(BaseModel):
 
 
 def test_history():
-    db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
     agent = Agent(
         model=Claude(id="claude-3-5-haiku-20241022"),
-        storage=PostgresAgentStorage(table_name="agent_sessions", db_url=db_url),
+        storage=SqliteAgentStorage(table_name="agent_sessions", db_file="tmp/agent_storage.db"),
         add_history_to_messages=True,
         telemetry=False,
         monitoring=False,

libs/agno/tests/integration/agent/__init__.py

@@ -222,5 +222,5 @@ def test_tool_call_list_parameters():
             tool_calls.extend(msg.tool_calls)
     for call in tool_calls:
         if call.get("type", "") == "function":
-            assert call["function"]["name"] == "get_contents"
+            assert call["function"]["name"] in ["get_contents", "exa_answer"]
     assert response.content is not None

libs/agno/tests/integration/knowledge/__init__.py

@@ -2,7 +2,7 @@
 
 from agno.agent import Agent, RunResponse  # noqa
 from agno.models.aws import AwsBedrock
-from agno.storage.agent.postgres import PostgresAgentStorage
+from agno.storage.agent.sqlite import SqliteAgentStorage
 
 
 def _assert_metrics(response: RunResponse):
@@ -72,17 +72,7 @@ def test_with_memory():
     assert [m.role for m in agent.memory.messages] == ["system", "user", "assistant", "user", "assistant"]
 
     # Test metrics structure and types
-    input_tokens = response2.metrics["input_tokens"]
-    output_tokens = response2.metrics["output_tokens"]
-    total_tokens = response2.metrics["total_tokens"]
-
-    assert isinstance(input_tokens[0], int)
-    assert input_tokens[0] > 0
-    assert isinstance(output_tokens[0], int)
-    assert output_tokens[0] > 0
-    assert isinstance(total_tokens[0], int)
-    assert total_tokens[0] > 0
-    assert total_tokens[0] == input_tokens[0] + output_tokens[0]
+    _assert_metrics(response2)
 
 
 def test_response_model():
@@ -109,10 +99,9 @@ class MovieScript(BaseModel):
 
 
 def test_history():
-    db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
     agent = Agent(
         model=AwsBedrock(id="anthropic.claude-3-sonnet-20240229-v1:0"),
-        storage=PostgresAgentStorage(table_name="agent_sessions", db_url=db_url),
+        storage=SqliteAgentStorage(table_name="agent_sessions", db_file="tmp/agent_storage.db"),
         add_history_to_messages=True,
         telemetry=False,
         monitoring=False,

libs/agno/tests/integration/models/__init__.py

@@ -173,5 +173,5 @@ def test_tool_call_list_parameters():
             tool_calls.extend(msg.tool_calls)
     for call in tool_calls:
         if call.get("type", "") == "function":
-            assert call["function"]["name"] == "get_contents"
+            assert call["function"]["name"] in ["get_contents", "exa_answer"]
     assert response.content is not None

libs/agno/tests/integration/models/anthropic/__init__.py

@@ -3,7 +3,7 @@
 
 from agno.agent import Agent, RunResponse  # noqa
 from agno.models.aws import Claude
-from agno.storage.agent.postgres import PostgresAgentStorage
+from agno.storage.agent.sqlite import SqliteAgentStorage
 
 
 def _assert_metrics(response: RunResponse):
@@ -101,17 +101,7 @@ def test_with_memory():
     assert [m.role for m in agent.memory.messages] == ["system", "user", "assistant", "user", "assistant"]
 
     # Test metrics structure and types
-    input_tokens = response2.metrics["input_tokens"]
-    output_tokens = response2.metrics["output_tokens"]
-    total_tokens = response2.metrics["total_tokens"]
-
-    assert isinstance(input_tokens[0], int)
-    assert input_tokens[0] > 0
-    assert isinstance(output_tokens[0], int)
-    assert output_tokens[0] > 0
-    assert isinstance(total_tokens[0], int)
-    assert total_tokens[0] > 0
-    assert total_tokens[0] == input_tokens[0] + output_tokens[0]
+    _assert_metrics(response2)
 
 
 def test_structured_output():
@@ -137,14 +127,12 @@ class MovieScript(BaseModel):
 
 
 def test_history():
-    db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
     agent = Agent(
         model=Claude(id="anthropic.claude-3-sonnet-20240229-v1:0"),
-        storage=PostgresAgentStorage(table_name="agent_sessions", db_url=db_url),
+        storage=SqliteAgentStorage(table_name="agent_sessions", db_file="tmp/agent_storage.db"),
         add_history_to_messages=True,
         telemetry=False,
         monitoring=False,
-        markdown=True,
     )
     agent.run("Hello")
     assert len(agent.run_response.messages) == 2

libs/agno/tests/integration/models/anthropic/test_basic.py

@@ -222,5 +222,5 @@ def test_tool_call_list_parameters():
             tool_calls.extend(msg.tool_calls)
     for call in tool_calls:
         if call.get("type", "") == "function":
-            assert call["function"]["name"] == "get_contents"
+            assert call["function"]["name"] in ["get_contents", "exa_answer"]
     assert response.content is not None

libs/agno/tests/integration/models/anthropic/test_tool_use.py

@@ -3,7 +3,7 @@
 
 from agno.agent import Agent, RunResponse
 from agno.models.azure import AzureAIFoundry
-from agno.storage.agent.postgres import PostgresAgentStorage
+from agno.storage.agent.sqlite import SqliteAgentStorage
 
 
 def _assert_metrics(response: RunResponse):
@@ -101,17 +101,7 @@ def test_with_memory():
     assert [m.role for m in agent.memory.messages] == ["system", "user", "assistant", "user", "assistant"]
 
     # Test metrics structure and types
-    input_tokens = response2.metrics["input_tokens"]
-    output_tokens = response2.metrics["output_tokens"]
-    total_tokens = response2.metrics["total_tokens"]
-
-    assert isinstance(input_tokens[0], int)
-    assert input_tokens[0] > 0
-    assert isinstance(output_tokens[0], int)
-    assert output_tokens[0] > 0
-    assert isinstance(total_tokens[0], int)
-    assert total_tokens[0] > 0
-    assert total_tokens[0] == input_tokens[0] + output_tokens[0]
+    _assert_metrics(response2)
 
 
 def test_response_model():
@@ -137,10 +127,9 @@ class MovieScript(BaseModel):
 
 
 def test_history():
-    db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
     agent = Agent(
         model=AzureAIFoundry(id="Phi-4"),
-        storage=PostgresAgentStorage(table_name="agent_sessions", db_url=db_url),
+        storage=SqliteAgentStorage(table_name="agent_sessions", db_file="tmp/agent_storage.db"),
         add_history_to_messages=True,
         telemetry=False,
         monitoring=False,

libs/agno/tests/integration/models/aws/__init__.py

@@ -222,5 +222,5 @@ def test_tool_call_list_parameters():
             tool_calls.extend(msg.tool_calls)
     for call in tool_calls:
         if call.get("type", "") == "function":
-            assert call["function"]["name"] == "get_contents"
+            assert call["function"]["name"] in ["get_contents", "exa_answer"]
     assert response.content is not None