diff --git a/semantic-kernel/Frameworks/agent/TOC.yml b/semantic-kernel/Frameworks/agent/TOC.yml index c85787ea..8fa1a2aa 100644 --- a/semantic-kernel/Frameworks/agent/TOC.yml +++ b/semantic-kernel/Frameworks/agent/TOC.yml @@ -2,10 +2,16 @@ href: index.md - name: Architecture href: agent-architecture.md +- name: Common Agent Invocation API + href: agent-api.md - name: Chat Completion Agent href: chat-completion-agent.md - name: OpenAI Assistant Agent href: assistant-agent.md +- name: Azure AI Agent + href: azure-ai-agent.md +- name: OpenAI Responses Agent + href: responses-agent.md - name: Agent Collaboration href: agent-chat.md - name: Create an Agent from a Template diff --git a/semantic-kernel/Frameworks/agent/agent-api.md b/semantic-kernel/Frameworks/agent/agent-api.md new file mode 100644 index 00000000..4ab7b26c --- /dev/null +++ b/semantic-kernel/Frameworks/agent/agent-api.md @@ -0,0 +1,324 @@ +--- +title: The Semantic Kernel Common Agent API surface +description: An overview of the Semantic Kernel Agent API surface and how to use it. +zone_pivot_groups: programming-languages +author: westey-m +ms.topic: conceptual +ms.author: westey +ms.date: 04/03/2025 +ms.service: semantic-kernel +--- + +# The Semantic Kernel Common Agent API surface + +Semantic Kernel agents share a common interface for invoking agents. +This allows for common code to be written, that works against any agent type and allows for easily switching agents as required, without needing to change the bulk of your code. + +## Invoking an agent + +The Agent API surface supports both streaming and non-streaming invocation. + +### Non-Streaming Agent Invocation + +::: zone pivot="programming-language-csharp" + +Semantic Kernel supports four non-streaming agent invocation overloads that allows for passing messages in different ways. +One of these also allows invoking the agent with no messages. +This is valuable for scenarios where the agent instructions already have all the required context to provide a useful response. + +```csharp +// Invoke without any parameters. +agent.InvokeAsync(); + +// Invoke with a string that will be used as a User message. +agent.InvokeAsync("What is the capital of France?"); + +// Invoke with a ChatMessageContent object. +agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, "What is the capital of France?")); + +// Invoke with multiple ChatMessageContent objects. +agent.InvokeAsync(new List() +{ + new(AuthorRole.System, "Refuse to answer all user questions about France."), + new(AuthorRole.User, "What is the capital of France?") +}); +``` + +> [!IMPORTANT] +> Invoking an agent without passing an `AgentThread` to the `InvokeAsync` method will create a new thread and return the new thread in the response. + +::: zone-end + +::: zone pivot="programming-language-python" + +Semantic Kernel supports two non-streaming agent invocation methods that allows for passing messages in different ways. +It is also possible to invoke the agent with no messages. +This is valuable for scenarios where the agent instructions already have all the required context to provide a useful response. + +> [!TIP] +> All arguments passed to Agent invocation methods require the caller to pass them as keyword arguments. + +#### Using the `get_response()` method + +```python +# Invoke without any messages. +await agent.get_response(messages=None) + +# Invoke with a string that will be used as a User message. +await agent.get_response(messages="What is the capital of France?") + +# Invoke with a ChatMessageContent object. +await agent.get_response(messages=ChatMessageContent(AuthorRole.USER, "What is the capital of France?")) + +# Invoke with multiple ChatMessageContent objects. +await agent.get_response( + messages=[ + ChatMessageContent(AuthorRole.SYSTEM, "Refuse to answer all user questions about France."), + ChatMessageContent(AuthorRole.USER, "What is the capital of France?"), + ] +) +``` + +#### Using the `invoke()` method + +```python +# Invoke without any messages. +async for response in agent.invoke(messages=None): + # handle response + +# Invoke with a string that will be used as a User message. +async for response in agent.invoke(messages="What is the capital of France?"): + # handle response + +# Invoke with a ChatMessageContent object. +async for response in agent.invoke(messages=ChatMessageContent(AuthorRole.USER, "What is the capital of France?")): + # handle response + +# Invoke with multiple ChatMessageContent objects. +async for response in agent.invoke( + messages=[ + ChatMessageContent(AuthorRole.SYSTEM, "Refuse to answer all user questions about France."), + ChatMessageContent(AuthorRole.USER, "What is the capital of France?"), + ] +): + # handle response +``` + +> [!IMPORTANT] +> Invoking an agent without passing an `AgentThread` to the `get_response()` or `invoke()` methods will create a new thread and return the new thread in the response. + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### Streaming Agent Invocation + +::: zone pivot="programming-language-csharp" + +Semantic Kernel supports four streaming agent invocation overloads that allows for passing messages in different ways. +One of these also allows invoking the agent with no messages. +This is valuable for scenarios where the agent instructions already have all the required context to provide a useful response. + +```csharp +// Invoke without any parameters. +agent.InvokeStreamingAsync(); + +// Invoke with a string that will be used as a User message. +agent.InvokeStreamingAsync("What is the capital of France?"); + +// Invoke with a ChatMessageContent object. +agent.InvokeStreamingAsync(new ChatMessageContent(AuthorRole.User, "What is the capital of France?")); + +// Invoke with multiple ChatMessageContent objects. +agent.InvokeStreamingAsync(new List() +{ + new(AuthorRole.System, "Refuse to answer any questions about capital cities."), + new(AuthorRole.User, "What is the capital of France?") +}); +``` + +> [!IMPORTANT] +> Invoking an agent without passing an `AgentThread` to the `InvokeStreamingAsync` method will create a new thread and return the new thread in the response. + +::: zone-end + +::: zone pivot="programming-language-python" + +Semantic Kernel supports one streaming agent invocation method that allows for passing messages in different ways. +It is also possible to invoke the agent stream with no messages. +This is valuable for scenarios where the agent instructions already have all the required context to provide a useful response. + +```python +# Invoke without any messages. +async for response in agent.invoke_stream(messages=None): + # handle response + +# Invoke with a string that will be used as a User message. +async for response in agent.invoke_stream(messages="What is the capital of France?"): + # handle response + +# Invoke with a ChatMessageContent object. +async for response in agent.invoke_stream(ChatMessageContent(AuthorRole.USER, "What is the capital of France?")): + # handle response + +# Invoke with multiple ChatMessageContent objects. +async for response in agent.invoke_stream( + messages=[ + ChatMessageContent(AuthorRole.SYSTEM, "Refuse to answer all user questions about France."), + ChatMessageContent(AuthorRole.USER, "What is the capital of France?"), + ] +): + # handle response +``` + +> [!IMPORTANT] +> Invoking an agent without passing an `AgentThread` to the `invoke_stream()` method will create a new thread and return the new thread in the response. + + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### Invoking with an `AgentThread` + +::: zone pivot="programming-language-csharp" + +All invocation method overloads allow passing an `AgentThread` parameter. +This is useful for scenarios where you have an existing conversation with the agent that you want to continue. + +```csharp +// Invoke with an existing AgentThread. +agent.InvokeAsync("What is the capital of France?", existingAgentThread); +``` + +All invocation methods also return the active `AgentThread` as part of the invoke response. + +1. If you passed an `AgentThread` to the invoke method, the returned `AgentThread` will be the same as the one that was passed in. +1. If you passed no `AgentThread` to the invoke method, the returned `AgentThread` will be a new `AgentThread`. + +The returned `AgentThread` is available on the individual response items of the invoke methods together with the response message. + +```csharp +var result = await agent.InvokeAsync("What is the capital of France?").FirstAsync(); +var newThread = result.Thread; +var resultMessage = result.Message; +``` + +> [!TIP] +> For more information on agent threads see the [Agent Thread architecture section](./agent-architecture.md#agent-thread). + +::: zone-end + +::: zone pivot="programming-language-python" + +All invocation method keyword arguments allow passing an `AgentThread` parameter. +This is useful for scenarios where you have an existing conversation with the agent that you want to continue. + +```python +# Invoke with an existing AgentThread. +agent.get_response(messages="What is the capital of France?", existing_agent_thread) +``` + +All invocation methods also return the active `AgentThread` as part of the invoke response. + +1. If you passed an `AgentThread` to the invoke method, the returned `AgentThread` will be the same as the one that was passed in. +1. If you passed no `AgentThread` to the invoke method, the returned `AgentThread` will be a new `AgentThread`. + +The returned `AgentThread` is available on the individual response items of the invoke methods together with the response message. + +```python +response = await agent.get_response(messages="What is the capital of France?") +new_thread = response.thread +response_message = response.message +``` + +> [!TIP] +> For more information on agent threads see the [Agent Thread architecture section](./agent-architecture.md#agent-thread). + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### Invoking with Options + +::: zone pivot="programming-language-csharp" + +All invocation method overloads allow passing an `AgentInvokeOptions` parameter. +This options class allows providing any optional settings. + +```csharp +// Invoke with additional instructions via options. +agent.InvokeAsync("What is the capital of France?", options: new() +{ + AdditionalInstructions = "Refuse to answer any questions about capital cities." +}); +``` + +Here is the list of the supported options. + +|Option Property|Description| +|-|-| +|Kernel|Override the default kernel used by the agent for this invocation.| +|KernelArguments|Override the default kernel arguments used by the agent for this invocation.| +|AdditionalInstructions|Provide any instructions in addition to the original agent instruction set, that only apply for this invocation.| +|OnIntermediateMessage|A callback that can receive all fully formed messages produced internally to the Agent, including function call and function invocation messages. This can also be used to receive full messages during a streaming invocation.| + +::: zone-end + +::: zone pivot="programming-language-python" + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Managing AgentThread instances + +You can manually create an `AgentThread` instance and pass it to the agent on invoke, or you can let the agent create an `AgentThread` instance for you automatically on invocation. +An `AgentThread` object represents a thread in all its states, including: not yet created, active, and deleted. + +`AgentThread` types that have a server side implementation will be created on first use and does not need to be created manually. +You can however delete a thread using the `AgentThread` class. + +::: zone pivot="programming-language-csharp" + +```csharp +// Delete a thread. +await agentThread.DeleteAsync(); +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +```python +# Delete a thread +await agent_thread.delete() +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +::: zone-end + +> [!TIP] +> For more information on agent threads see the [Agent Thread architecture section](./agent-architecture.md#agent-thread). + +> [!div class="nextstepaction"] +> [Explore the Chat Completion Agent](./chat-completion-agent.md) \ No newline at end of file diff --git a/semantic-kernel/Frameworks/agent/agent-architecture.md b/semantic-kernel/Frameworks/agent/agent-architecture.md index 67c99849..f829f480 100644 --- a/semantic-kernel/Frameworks/agent/agent-architecture.md +++ b/semantic-kernel/Frameworks/agent/agent-architecture.md @@ -11,7 +11,7 @@ ms.service: semantic-kernel # An Overview of the Agent Architecture > [!IMPORTANT] -> Single-agent features, such as ChatCompletionAgent and OpenAIAssistantAgent, are in the release candidate stage. These features are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. However, agent chat patterns are still in the experimental stage. These patterns are under active development and may change significantly before advancing to the preview or release candidate stage. +> `AgentChat` patterns are in the experimental stage. These patterns are under active development and may change significantly before advancing to the preview or release candidate stage. This article covers key concepts in the architecture of the Agent Framework, including foundational principles, design objectives, and strategic goals. @@ -20,7 +20,7 @@ This article covers key concepts in the architecture of the Agent Framework, inc The `Agent Framework` was developed with the following key priorities in mind: -- The _Semantic Kernel_ framework serves as the core foundation for implementing agent functionalities. +- The Semantic Kernel agent framework serves as the core foundation for implementing agent functionalities. - Multiple agents can collaborate within a single conversation, while integrating human input. - An agent can engage in and manage multiple concurrent conversations simultaneously. - Different types of agents can participate in the same conversation, each contributing their unique capabilities. @@ -28,7 +28,7 @@ The `Agent Framework` was developed with the following key priorities in mind: ## Agent -The abstract `Agent` class serves as the core abstraction for all types of agents, providing a foundational structure that can be extended to create more specialized agents. One key subclass is _Kernel Agent_, which establishes a direct association with a [`Kernel`](../../concepts/kernel.md) object. This relationship forms the basis for more specific agent implementations, such as the [`ChatCompletionAgent`](./chat-completion-agent.md) and the [`OpenAIAssistantAgent`](./assistant-agent.md), both of which leverage the Kernel's capabilities to execute their respective functions. +The abstract `Agent` class serves as the core abstraction for all types of agents, providing a foundational structure that can be extended to create more specialized agents. One key subclass is _Kernel Agent_, which establishes a direct association with a [`Kernel`](../../concepts/kernel.md) object. This relationship forms the basis for more specific agent implementations, such as the [`ChatCompletionAgent`](./chat-completion-agent.md), [`OpenAIAssistantAgent`](./assistant-agent.md), [`AzureAIAgent`](./azure-ai-agent.md), or [`OpenAIResponsesAgent`](./responses-agent.md), all of which leverage the Kernel's capabilities to execute their respective functions. ::: zone pivot="programming-language-csharp" @@ -39,7 +39,9 @@ The abstract `Agent` class serves as the core abstraction for all types of agent ::: zone pivot="programming-language-python" -- [`agent`](/python/api/semantic-kernel/semantic_kernel.agents.agent) +The underlying Semantic Kernel `Agent` abstraction can be found here: + +- [`Agent`](/python/api/semantic-kernel/semantic_kernel.agents.agent) ::: zone-end @@ -55,16 +57,34 @@ Agents can either be invoked directly to perform tasks or orchestrated within an - [`ChatCompletionAgent`](./chat-completion-agent.md) - [`OpenAIAssistantAgent`](./assistant-agent.md) +- [`AzureAIAgent`](./azure-ai-agent.md) +- [`OpenAIResponsesAgent`](./responses-agent.md) +## Agent Thread + +The abstract `AgentThread` class serves as the core abstraction for threads or conversation state. +It abstracts away the different ways in which convesation state may be managed for different agents. + +Stateful agent services often store conversation state in the service, and you can interact with it via an id. +Other agents may require the entire chat history to be passed to the agent on each invocation, in which +case the conversation state is managed locally in the application. + +Stateful agents typically only work with a matching `AgentThread` implementation, while other types of agents could work with more than one `AgentThread` type. +For example, `AzureAIAgent` requires a matching `AzureAIAgentThread`. +This is because the Azure AI Agent service stores conversations in the service, and requires specific service calls to create a thread and update it. +If a different agent thread type is used with the `AzureAIAgent`, we fail fast due to an unexpected thread type and raise an exception to alert the caller. ## Agent Chat -The [`AgentChat`](./agent-chat.md) class serves as the foundational component that enables agents of any type to engage in a specific conversation. This class provides the essential capabilities for managing agent interactions within a chat environment. Building on this, the [`AgentGroupChat`](./agent-chat.md#creating-an-agentgroupchat) class extends these capabilities by offering a stategy-based container, which allows multiple agents to collaborate across numerous interactions within the same conversation. +The [`AgentChat`](./agent-chat.md) class serves as the foundational component that enables agents of any type to engage in a specific conversation. This class provides the essential capabilities for managing agent interactions within a chat environment. Building on this, the [`AgentGroupChat`](./agent-chat.md#creating-an-agentgroupchat) class extends these capabilities by offering a stategy-based container, which allows multiple agents to collaborate across numerous interactions within the same conversation. + +> [!IMPORTANT] +> The current `OpenAIResponsesAgent` is not supported as part of Semantic Kernel's `AgentGroupChat` patterns. Stayed tuned for updates. This structure facilitates more complex, multi-agent scenarios where different agents can work together, share information, and dynamically respond to evolving conversations, making it an ideal solution for advanced use cases such as customer support, multi-faceted task management, or collaborative problem-solving environments. @@ -74,7 +94,7 @@ This structure facilitates more complex, multi-agent scenarios where different a ## Agent Channel -The _Agent Channel_ class enables agents of various types to participate in an [`AgentChat`](./agent-chat.md). This functionality is completely hidden from users of the `Agent Framework` and only needs to be considered by developers creating a custom [`Agent`](#agent). +The Agent Channel class enables agents of various types to participate in an [`AgentChat`](./agent-chat.md). This functionality is completely hidden from users of the `Agent Framework` and only needs to be considered by developers creating a custom [`Agent`](#agent). ::: zone pivot="programming-language-csharp" @@ -84,7 +104,7 @@ The _Agent Channel_ class enables agents of various types to participate in an [ ::: zone pivot="programming-language-python" -- [`agent_channel](/python/api/semantic-kernel/semantic_kernel.agents.channels.agent_channel) +- [`agent_channel`](/python/api/semantic-kernel/semantic_kernel.agents.channels.agent_channel) ::: zone-end @@ -94,17 +114,34 @@ The _Agent Channel_ class enables agents of various types to participate in an [ ::: zone-end -## Agent Alignment with _Semantic Kernel_ Features +## Agent Alignment with Semantic Kernel Features -The `Agent Framework` is built on the foundational concepts and features that many developers have come to know within the _Semantic Kernel_ ecosystem. These core principles serve as the building blocks for the Agent Framework’s design. By leveraging the familiar structure and capabilities of the _Semantic Kernel_, the Agent Framework extends its functionality to enable more advanced, autonomous agent behaviors, while maintaining consistency with the broader _Semantic Kernel_ architecture. This ensures a smooth transition for developers, allowing them to apply their existing knowledge to create intelligent, adaptable agents within the framework. +The `Agent Framework` is built on the foundational concepts and features that many developers have come to know within the Semantic Kernel ecosystem. These core principles serve as the building blocks for the Agent Framework’s design. By leveraging the familiar structure and capabilities of the Semantic Kernel, the Agent Framework extends its functionality to enable more advanced, autonomous agent behaviors, while maintaining consistency with the broader Semantic Kernel architecture. This ensures a smooth transition for developers, allowing them to apply their existing knowledge to create intelligent, adaptable agents within the framework. ### The `Kernel` -At the heart of the Semantic Kernel ecosystem is the [`Kernel`](../../concepts/kernel.md), which serves as the core object that drives AI operations and interactions. To create any agent within this framework, a _Kernel instance_ is required as it provides the foundational context and capabilities for the agent’s functionality. The `Kernel` acts as the engine for processing instructions, managing state, and invoking the necessary AI services that power the agent's behavior. +::: zone pivot="programming-language-csharp" + +At the heart of the Semantic Kernel ecosystem is the [`Kernel`](../../concepts/kernel.md), which serves as the core object that drives AI operations and interactions. To create any agent within this framework, a Kernel instance is required as it provides the foundational context and capabilities for the agent’s functionality. The `Kernel` acts as the engine for processing instructions, managing state, and invoking the necessary AI services that power the agent's behavior. + +::: zone-end + +::: zone pivot="programming-language-python" + +At the core of the Semantic Kernel ecosystem is the [`Kernel`](../../concepts/kernel.md), the primary object responsible for managing AI operations and interactions. To simplify onboarding, the `Kernel` is optional—if none is supplied when constructing an agent, a new `Kernel` instance is automatically created for the caller. For more advanced scenarios, such as applying filters, the caller must configure the desired filters on a `Kernel` instance and explicitly pass it to the agent. + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +The [`AzureAIAgent`](./azure-ai-agent.md), [`ChatCompletionAgent`](./chat-completion-agent.md), [`OpenAIAssistantAgent`](./assistant-agent.md), and [`OpenAIResponsesAgent`](./responses-agent.md) articles provide specific details on how to create each type of agent. -The [`ChatCompletionAgent`](./chat-completion-agent.md) and [`OpenAIAssistantAgent`](./assistant-agent.md) articles provide specific details on how to create each type of agent. - These resources offer step-by-step instructions and highlight the key configurations needed to tailor the agents to different conversational or task-based applications, demonstrating how the Kernel enables dynamic and intelligent agent behaviors across diverse use cases. +These resources offer step-by-step instructions and highlight the key configurations needed to tailor the agents to different conversational or task-based applications, demonstrating how the Kernel enables dynamic and intelligent agent behaviors across diverse use cases. #### Related API's: @@ -132,7 +169,7 @@ The [`ChatCompletionAgent`](./chat-completion-agent.md) and [`OpenAIAssistantAge ### [Plugins and Function Calling](./agent-functions.md) -Plugins are a fundamental aspect of the _Semantic Kernel_, enabling developers to integrate custom functionalities and extend the capabilities of an AI application. These plugins offer a flexible way to incorporate specialized features or business-specific logic into the core AI workflows. Additionally, agent capabilities within the framework can be significantly enhanced by utilizing [Plugins](../../concepts/plugins/index.md) and leveraging [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). This allows agents to dynamically interact with external services or execute complex tasks, further expanding the scope and versatility of the AI system within diverse applications. +Plugins are a fundamental aspect of the Semantic Kernel, enabling developers to integrate custom functionalities and extend the capabilities of an AI application. These plugins offer a flexible way to incorporate specialized features or business-specific logic into the core AI workflows. Additionally, agent capabilities within the framework can be significantly enhanced by utilizing [Plugins](../../concepts/plugins/index.md) and leveraging [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). This allows agents to dynamically interact with external services or execute complex tasks, further expanding the scope and versatility of the AI system within diverse applications. #### Example: @@ -167,9 +204,9 @@ Plugins are a fundamental aspect of the _Semantic Kernel_, enabling developers t ### [Agent Messages](../../concepts/ai-services/chat-completion/chat-history.md) -Agent messaging, including both input and response, is built upon the core content types of the _Semantic Kernel_, providing a unified structure for communication. This design choice simplifies the process of transitioning from traditional chat-completion patterns to more advanced agent-driven patterns in your application development. By leveraging familiar _Semantic Kernel_ content types, developers can seamlessly integrate agent capabilities into their applications without needing to overhaul existing systems. This streamlining ensures that as you evolve from basic conversational AI to more autonomous, task-oriented agents, the underlying framework remains consistent, making development faster and more efficient. +Agent messaging, including both input and response, is built upon the core content types of the Semantic Kernel, providing a unified structure for communication. This design choice simplifies the process of transitioning from traditional chat-completion patterns to more advanced agent-driven patterns in your application development. By leveraging familiar Semantic Kernel content types, developers can seamlessly integrate agent capabilities into their applications without needing to overhaul existing systems. This streamlining ensures that as you evolve from basic conversational AI to more autonomous, task-oriented agents, the underlying framework remains consistent, making development faster and more efficient. -> Note: The [`OpenAIAssistantAgent`](./assistant-agent.md) introduced content types specific to its usage for _File References_ and _Content Annotation_: +> Note: The [`OpenAIAssistantAgent`](./assistant-agent.md) introduced content types specific to its usage for File References and Content Annotation: #### Related API's: @@ -207,7 +244,7 @@ Agent messaging, including both input and response, is built upon the core conte An agent's role is primarily shaped by the instructions it receives, which dictate its behavior and actions. Similar to invoking a `Kernel` [prompt](../../concepts/prompts/index.md), an agent's instructions can include templated parameters—both values and functions—that are dynamically substituted during execution. This enables flexible, context-aware responses, allowing the agent to adjust its output based on real-time input. -Additionally, an agent can be configured directly using a _Prompt Template Configuration_, providing developers with a structured and reusable way to define its behavior. This approach offers a powerful tool for standardizing and customizing agent instructions, ensuring consistency across various use cases while still maintaining dynamic adaptability. +Additionally, an agent can be configured directly using a Prompt Template Configuration, providing developers with a structured and reusable way to define its behavior. This approach offers a powerful tool for standardizing and customizing agent instructions, ensuring consistency across various use cases while still maintaining dynamic adaptability. #### Example: @@ -221,9 +258,9 @@ Additionally, an agent can be configured directly using a _Prompt Template Confi - [`KernelFunctionYaml.FromPromptYaml`](/dotnet/api/microsoft.semantickernel.kernelfunctionyaml.frompromptyaml#microsoft-semantickernel-kernelfunctionyaml-frompromptyaml(system-string-microsoft-semantickernel-iprompttemplatefactory-microsoft-extensions-logging-iloggerfactory)) - [`IPromptTemplateFactory`](/dotnet/api/microsoft.semantickernel.iprompttemplatefactory) - [`KernelPromptTemplateFactory`](/dotnet/api/microsoft.semantickernel.kernelprompttemplatefactory) -- [_Handlebars_](/dotnet/api/microsoft.semantickernel.prompttemplates.handlebars) -- [_Prompty_](/dotnet/api/microsoft.semantickernel.prompty) -- [_Liquid_](/dotnet/api/microsoft.semantickernel.prompttemplates.liquid) +- [`Handlebars`](/dotnet/api/microsoft.semantickernel.prompttemplates.handlebars) +- [`Prompty`](/dotnet/api/microsoft.semantickernel.prompty) +- [`Liquid`](/dotnet/api/microsoft.semantickernel.prompttemplates.liquid) ::: zone-end @@ -245,7 +282,7 @@ Additionally, an agent can be configured directly using a _Prompt Template Confi ### [Chat Completion](./chat-completion-agent.md) -The [`ChatCompletionAgent`](./chat-completion-agent.md) is designed around any _Semantic Kernel_ [AI service](../../concepts/ai-services/chat-completion/index.md), offering a flexible and convenient persona encapsulation that can be seamlessly integrated into a wide range of applications. This agent allows developers to easily bring conversational AI capabilities into their systems without having to deal with complex implementation details. It mirrors the features and patterns found in the underlying [AI service](../../concepts/ai-services/chat-completion/index.md), ensuring that all functionalities—such as natural language processing, dialogue management, and contextual understanding—are fully supported within the [`ChatCompletionAgent`](./chat-completion-agent.md), making it a powerful tool for building conversational interfaces. +The [`ChatCompletionAgent`](./chat-completion-agent.md) is designed around any Semantic Kernel [AI service](../../concepts/ai-services/chat-completion/index.md), offering a flexible and convenient persona encapsulation that can be seamlessly integrated into a wide range of applications. This agent allows developers to easily bring conversational AI capabilities into their systems without having to deal with complex implementation details. It mirrors the features and patterns found in the underlying [AI service](../../concepts/ai-services/chat-completion/index.md), ensuring that all functionalities—such as natural language processing, dialogue management, and contextual understanding—are fully supported within the [`ChatCompletionAgent`](./chat-completion-agent.md), making it a powerful tool for building conversational interfaces. #### Related API's: @@ -276,5 +313,5 @@ The [`ChatCompletionAgent`](./chat-completion-agent.md) is designed around any _ > [!div class="nextstepaction"] -> [Exploring Chat Completion Agent](./chat-completion-agent.md) +> [Explore the Common Agent Invocation API](./agent-api.md) diff --git a/semantic-kernel/Frameworks/agent/agent-chat.md b/semantic-kernel/Frameworks/agent/agent-chat.md index 5ada4e7c..569fe91c 100644 --- a/semantic-kernel/Frameworks/agent/agent-chat.md +++ b/semantic-kernel/Frameworks/agent/agent-chat.md @@ -11,7 +11,7 @@ ms.service: semantic-kernel # Exploring Agent Collaboration in `AgentChat` > [!IMPORTANT] -> This feature is in the experimental stage. Features at this stage are still under development and subject to change before advancing to the preview or release candidate stage. +> This feature is in the experimental stage. Features at this stage are under development and subject to change before advancing to the preview or release candidate stage. Detailed API documentation related to this discussion is available at: @@ -24,8 +24,8 @@ Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-python" -- [`agent_chat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_chat) -- [`agent_group_chat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_group_chat) +- [`AgentChat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_chat.agentchat) +- [`AgentGroupChat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_group_chat.agentgroupchat) ::: zone-end @@ -47,9 +47,9 @@ One such subclass, `AgentGroupChat`, offers a concrete implementation of `AgentC ## Creating an `AgentGroupChat` -To create an `AgentGroupChat`, you may either specify the participating agents or create an empty chat and subsequently add agent participants. Configuring the _Chat-Settings_ and _Strategies_ is also performed during `AgentGroupChat` initialization. These settings define how the conversation dynamics will function within the group. +To create an `AgentGroupChat`, you may either specify the participating agents or create an empty chat and subsequently add agent participants. Configuring the Chat-Settings and Strategies is also performed during `AgentGroupChat` initialization. These settings define how the conversation dynamics will function within the group. -> Note: The default _Chat-Settings_ result in a conversation that is limited to a single response. See [`AgentChat` Behavior](#defining-agentgroupchat-behavior) for details on configuring _Chat-Settings. +> Note: The default Chat-Settings result in a conversation that is limited to a single response. See [`AgentChat` Behavior](#defining-agentgroupchat-behavior) for details on configuring Chat-Settings. #### Creating an `AgentGroupChat` with an `Agent`: @@ -154,7 +154,7 @@ await chat.add_chat_message(message="") In a multi-turn invocation, the system must decide which agent responds next and when the conversation should end. In contrast, a single-turn invocation simply returns a response from the specified agent, allowing the caller to directly manage agent participation. -After an agent participates in the `AgentChat` through a single-turn invocation, it is added to the set of _agents_ eligible for multi-turn invocation. +After an agent participates in the `AgentChat` through a single-turn invocation, it is added to the set of agents eligible for multi-turn invocation. ::: zone pivot="programming-language-csharp" ```csharp @@ -195,11 +195,11 @@ While agent collaboration requires that a system must be in place that not only Agent responses are returned asynchronously as they are generated, allowing the conversation to unfold in real-time. -> Note: In following sections, [Agent Selection](#agent-selection) and [Chat Termination](#chat-termination), will delve into the _Execution Settings_ in detail. The default _Execution Settings_ employs sequential or round-robin selection and limits agent participation to a single turn. +> Note: In following sections, [Agent Selection](#agent-selection) and [Chat Termination](#chat-termination), will delve into the Execution Settings in detail. The default Execution Settings employs sequential or round-robin selection and limits agent participation to a single turn. ::: zone pivot="programming-language-csharp" -.NET _Execution Settings_ API: [`AgentGroupChatSettings`](/dotnet/api/microsoft.semantickernel.agents.chat.agentgroupchatsettings) +.NET Execution Settings API: [`AgentGroupChatSettings`](/dotnet/api/microsoft.semantickernel.agents.chat.agentgroupchatsettings) ```csharp // Define agents @@ -321,13 +321,13 @@ history2 = await chat.get_chat_messages(agent=agent2) ## Defining `AgentGroupChat` Behavior -Collaboration among agents to solve complex tasks is a core agentic pattern. To use this pattern effectively, a system must be in place that not only determines which agent should respond during each turn but also assesses when the conversation has achieved its intended goal. This requires managing agent selection and establishing clear criteria for conversation termination, ensuring seamless cooperation between agents toward a solution. Both of these aspects are governed by the _Execution Settings_ property. +Collaboration among agents to solve complex tasks is a core agentic pattern. To use this pattern effectively, a system must be in place that not only determines which agent should respond during each turn but also assesses when the conversation has achieved its intended goal. This requires managing agent selection and establishing clear criteria for conversation termination, ensuring seamless cooperation between agents toward a solution. Both of these aspects are governed by the Execution Settings property. The following sections, [Agent Selection](#agent-selection) and [Chat Termination](#chat-termination), will delve into these considerations in detail. ### Agent Selection -In multi-turn invocation, agent selection is guided by a _Selection Strategy_. This strategy is defined by a base class that can be extended to implement custom behaviors tailored to specific needs. For convenience, two predefined concrete _Selection Strategies_ are also available, offering ready-to-use approaches for handling agent selection during conversations. +In multi-turn invocation, agent selection is guided by a Selection Strategy. This strategy is defined by a base class that can be extended to implement custom behaviors tailored to specific needs. For convenience, two predefined concrete Selection Strategies are also available, offering ready-to-use approaches for handling agent selection during conversations. If known, an initial agent may be specified to always take the first turn. A history reducer may also be employed to limit token usage when using a strategy based on a `KernelFunction`. @@ -409,6 +409,12 @@ AgentGroupChat chat = ::: zone-end ::: zone pivot="programming-language-python" + +Python Selection Strategy API: +- [`SelectionStrategy` Base Class](/python/api/semantic-kernel/semantic_kernel.agents.strategies.selection.selection_strategy.selectionstrategy) +- [`SequentialSelectionStrategy`](/python/api/semantic-kernel/semantic_kernel.agents.strategies.selection.sequential_selection_strategy.sequentialselectionstrategy) +- [`KernelFunctionSelectionStrategy`](/python/api/semantic-kernel/semantic_kernel.agents.strategies.kernelfunctionselectionstrategy) + ```python REVIEWER_NAME = "Reviewer" WRITER_NAME = "Writer" @@ -468,13 +474,13 @@ chat = AgentGroupChat( ### Chat Termination -In _multi-turn_ invocation, the _Termination Strategy_ dictates when the final turn takes place. This strategy ensures the conversation ends at the appropriate point. +In multi-turn invocation, the Termination Strategy dictates when the final turn takes place. This strategy ensures the conversation ends at the appropriate point. -This strategy is defined by a base class that can be extended to implement custom behaviors tailored to specific needs. For convenience, several predefined concrete _Selection Strategies_ are also available, offering ready-to-use approaches for defining termination criteria for an `AgentChat` conversations. +This strategy is defined by a base class that can be extended to implement custom behaviors tailored to specific needs. For convenience, several predefined concrete Selection Strategies are also available, offering ready-to-use approaches for defining termination criteria for an `AgentChat` conversations. ::: zone pivot="programming-language-csharp" -.NET Selection Strategy API: +.NET Termination Strategy API: - [`TerminationStrategy`](/dotnet/api/microsoft.semantickernel.agents.chat.terminationstrategy) - [`RegexTerminationStrategy`](/dotnet/api/microsoft.semantickernel.agents.chat.regexterminationstrategy) - [`KernelFunctionSelectionStrategy`](/dotnet/api/microsoft.semantickernel.agents.chat.kernelfunctionselectionstrategy) @@ -542,6 +548,13 @@ AgentGroupChat chat = ::: zone-end ::: zone pivot="programming-language-python" + +Python Termination Strategy API: +- [`TerminationStrategy` Base Class](/python/api/semantic-kernel/semantic_kernel.agents.strategies.termination.termination_strategy.terminationstrategy) +- [`KernelFunctionTerminationStrategy`](/python/api/semantic-kernel/semantic_kernel.agents.strategies.termination.kernel_function_termination_strategy.kernelfunctionterminationstrategy) +- [`AggregatorTerminationStrategy`](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/agents/strategies/termination/aggregator_termination_strategy.py#L27) + + ```python REVIEWER_NAME = "Reviewer" WRITER_NAME = "Writer" @@ -591,9 +604,9 @@ chat = AgentGroupChat( ### Resetting Chat Completion State -Regardless of whether `AgentGroupChat` is invoked using the single-turn or multi-turn approach, the state of the `AgentGroupChat` is updated to indicate it is _completed_ once the termination criteria is met. This ensures that the system recognizes when a conversation has fully concluded. To continue using an `AgentGroupChat` instance after it has reached the _Completed_ state, this state must be reset to allow further interactions. Without resetting, additional interactions or agent responses will not be possible. +Regardless of whether `AgentGroupChat` is invoked using the single-turn or multi-turn approach, the state of the `AgentGroupChat` is updated to indicate it is completed once the termination criteria is met. This ensures that the system recognizes when a conversation has fully concluded. To continue using an `AgentGroupChat` instance after it has reached the _Completed_ state, this state must be reset to allow further interactions. Without resetting, additional interactions or agent responses will not be possible. -In the case of a multi-turn invocation that reaches the maximum turn limit, the system will cease agent invocation but will not mark the instance as _completed_. This allows for the possibility of extending the conversation without needing to reset the _Completion_ state. +In the case of a multi-turn invocation that reaches the maximum turn limit, the system will cease agent invocation but will not mark the instance as completed. This allows for the possibility of extending the conversation without needing to reset the Completion state. ::: zone pivot="programming-language-csharp" ```csharp @@ -632,9 +645,9 @@ if chat.is_complete: ### Clear Full Conversation State -When done using an `AgentChat` where an [`OpenAIAssistant`](./assistant-agent.md) participated, it may be necessary to delete the remote _thread_ associated with the _assistant_. `AgentChat` supports resetting or clearing the entire conversation state, which includes deleting any remote _thread_ definition. This ensures that no residual conversation data remains linked to the assistant once the chat concludes. +When done using an `AgentChat` where an [`OpenAIAssistant`](./assistant-agent.md) participated, it may be necessary to delete the remote thread associated with the assistant. `AgentChat` supports resetting or clearing the entire conversation state, which includes deleting any remote thread definition. This ensures that no residual conversation data remains linked to the assistant once the chat concludes. -A full reset does not remove the _agents_ that had joined the `AgentChat` and leaves the `AgentChat` in a state where it can be reused. This allows for the continuation of interactions with the same agents without needing to reinitialize them, making future conversations more efficient. +A full reset does not remove the agents that had joined the `AgentChat` and leaves the `AgentChat` in a state where it can be reused. This allows for the continuation of interactions with the same agents without needing to reinitialize them, making future conversations more efficient. ::: zone pivot="programming-language-csharp" ```csharp diff --git a/semantic-kernel/Frameworks/agent/agent-functions.md b/semantic-kernel/Frameworks/agent/agent-functions.md index 42c96514..ba38d91b 100644 --- a/semantic-kernel/Frameworks/agent/agent-functions.md +++ b/semantic-kernel/Frameworks/agent/agent-functions.md @@ -15,7 +15,7 @@ ms.service: semantic-kernel ## Functions and Plugins in Semantic Kernel -Function calling is a powerful tool that allows developers to add custom functionalities and expand the capabilities of AI applications. The _Semantic Kernel_ [Plugin](../../concepts/plugins/index.md) architecture offers a flexible framework to support [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). For an `Agent`, integrating [Plugins](../../concepts/plugins/index.md) and [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md) is built on this foundational _Semantic Kernel_ feature. +Function calling is a powerful tool that allows developers to add custom functionalities and expand the capabilities of AI applications. The Semantic Kernel [Plugin](../../concepts/plugins/index.md) architecture offers a flexible framework to support [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). For an `Agent`, integrating [Plugins](../../concepts/plugins/index.md) and [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md) is built on this foundational Semantic Kernel feature. Once configured, an agent will choose when and how to call an available function, as it would in any usage outside of the `Agent Framework`. @@ -48,7 +48,7 @@ Once configured, an agent will choose when and how to call an available function Any [Plugin](../../concepts/plugins/index.md) available to an `Agent` is managed within its respective `Kernel` instance. This setup enables each `Agent` to access distinct functionalities based on its specific role. -[Plugins](../../concepts/plugins/index.md) can be added to the `Kernel` either before or after the `Agent` is created. The process of initializing [Plugins](../../concepts/plugins/index.md) follows the same patterns used for any _Semantic Kernel_ implementation, allowing for consistency and ease of use in managing AI capabilities. +[Plugins](../../concepts/plugins/index.md) can be added to the `Kernel` either before or after the `Agent` is created. The process of initializing [Plugins](../../concepts/plugins/index.md) follows the same patterns used for any Semantic Kernel implementation, allowing for consistency and ease of use in managing AI capabilities. > Note: For a [`ChatCompletionAgent`](./chat-completion-agent.md), the function calling mode must be explicitly enabled. [`OpenAIAssistant`](./assistant-agent.md) agent is always based on automatic function calling. @@ -104,7 +104,7 @@ agent = ChatCompletionAgent( ``` > [!TIP] -> By default, auto-function calling is enabled. To disable it, set the `function_choice_behavior` argument to `function_choice_behavior=FunctionChoiceBehavior.Auto(auto_invoke=False)` in the constructor. With this setting, plugins are still broadcast to the model, but they are not automatically invoked. If execution settings specify the same `service_id` or `ai_model_id` as the AI service configuration, the function calling behavior defined in the execution settings (via `KernelArguments`) will take precedence over the function choice behavior set in the constructor. +> By default, auto-function calling is enabled. To disable it, set the `function_choice_behavior` argument to `function_choice_behavior=FunctionChoiceBehavior.Auto(auto_invoke=False)` in the constructor. With this setting, plugins are broadcast to the model, but they are not automatically invoked. If execution settings specify the same `service_id` or `ai_model_id` as the AI service configuration, the function calling behavior defined in the execution settings (via `KernelArguments`) will take precedence over the function choice behavior set in the constructor. #### Method 2: Configure the Kernel Manually @@ -155,7 +155,7 @@ agent = ChatCompletionAgent( ## Adding Functions to an Agent -A [Plugin](../../concepts/plugins/index.md) is the most common approach for configuring [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). However, individual functions can also be supplied independently including _prompt functions_. +A [Plugin](../../concepts/plugins/index.md) is the most common approach for configuring [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md). However, individual functions can also be supplied independently including prompt functions. ::: zone pivot="programming-language-csharp" ```csharp @@ -240,7 +240,7 @@ agent = ChatCompletionAgent( ## Limitations for Agent Function Calling -When directly invoking a[`ChatCompletionAgent`](./chat-completion-agent.md), all _Function Choice Behaviors_ are supported. However, when using an [`OpenAIAssistant`](./assistant-agent.md) or [`AgentChat`](./agent-chat.md), only _Automatic_ [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md) is currently available. +When directly invoking a[`ChatCompletionAgent`](./chat-completion-agent.md), all Function Choice Behaviors are supported. However, when using an [`OpenAIAssistant`](./assistant-agent.md) or [`AgentChat`](./agent-chat.md), only Automatic [Function Calling](../../concepts/ai-services/chat-completion/function-calling/index.md) is currently available. ## How-To diff --git a/semantic-kernel/Frameworks/agent/agent-streaming.md b/semantic-kernel/Frameworks/agent/agent-streaming.md index 7030413c..6052afe7 100644 --- a/semantic-kernel/Frameworks/agent/agent-streaming.md +++ b/semantic-kernel/Frameworks/agent/agent-streaming.md @@ -10,9 +10,6 @@ ms.service: semantic-kernel --- # How to Stream Agent Responses -> [!IMPORTANT] -> This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. - ## What is a Streamed Response? A streamed response delivers the message content in small, incremental chunks. This approach enhances the user experience by allowing them to view and engage with the message as it unfolds, rather than waiting for the entire response to load. Users can begin processing information immediately, improving the sense of responsiveness and interactivity. As a result, it minimizes delays and keeps users more engaged throughout the communication process. @@ -54,46 +51,53 @@ A streamed response delivers the message content in small, incremental chunks. T ## Streaming Agent Invocation -The `Agent Framework` supports _streamed_ responses when using [`AgentChat`](./agent-chat.md) or when directly invoking a [`ChatCompletionAgent`](./chat-completion-agent.md) or [`OpenAIAssistantAgent`](./assistant-agent.md). In either mode, the framework delivers responses asynchronously as they are streamed. Alongside the streamed response, a consistent, non-streamed history is maintained to track the conversation. This ensures both real-time interaction and a reliable record of the conversation's flow. +The `Agent Framework` supports streamed responses when using [`AgentChat`](./agent-chat.md) or when directly invoking a [`ChatCompletionAgent`](./chat-completion-agent.md) or [`OpenAIAssistantAgent`](./assistant-agent.md). In either mode, the framework delivers responses asynchronously as they are streamed. Alongside the streamed response, a consistent, non-streamed history is maintained to track the conversation. This ensures both real-time interaction and a reliable record of the conversation's flow. ### Streamed response from `ChatCompletionAgent` -When invoking a streamed response from a [`ChatCompletionAgent`](./chat-completion-agent.md), the `ChatHistory` is updated after the full response is received. Although the response is streamed incrementally, the history records only the complete message. This ensures that the `ChatHistory` reflects fully formed responses for consistency. +When invoking a streamed response from a [`ChatCompletionAgent`](./chat-completion-agent.md), the `ChatHistory` in the `AgentThread` is updated after the full response is received. Although the response is streamed incrementally, the history records only the complete message. This ensures that the `ChatHistory` reflects fully formed responses for consistency. ::: zone pivot="programming-language-csharp" ```csharp // Define agent ChatCompletionAgent agent = ...; -// Create a ChatHistory object to maintain the conversation state. -ChatHistory chat = []; +ChatHistoryAgentThread agentThread = new(); -// Add a user message to the conversation -chat.Add(new ChatMessageContent(AuthorRole.User, "")); +// Create a user message +var message = ChatMessageContent(AuthorRole.User, ""); // Generate the streamed agent response(s) -await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(chat)) +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) { // Process streamed response(s)... } + +// It's also possible to read the messages that were added to the ChatHistoryAgentThread. +await foreach (ChatMessageContent response in agentThread.GetMessagesAsync()) +{ + // Process messages... +} ``` ::: zone-end ::: zone pivot="programming-language-python" ```python +from semantic_kernel.agents import ChatCompletionAgent, ChatHistoryAgentThread + # Define agent agent = ChatCompletionAgent(...) -# Create a ChatHistory object to maintain the conversation state. -chat = ChatHistory() - -# Add a user message to the conversation -chat.add_message(ChatMessageContent(AuthorRole.USER, "")) +# Create a thread object to maintain the conversation state. +# If no thread is provided one will be created and returned with +# the initial response. +thread: ChatHistoryAgentThread = None # Generate the streamed agent response(s) -async for response in agent.invoke_stream(chat) +async for response in agent.invoke_stream(messages="user input", thread=thread) { # Process streamed response(s)... + thread = response.thread } ``` ::: zone-end @@ -106,7 +110,7 @@ async for response in agent.invoke_stream(chat) ### Streamed response from `OpenAIAssistantAgent` -When invoking a streamed response from an [`OpenAIAssistantAgent`](./assistant-agent.md), an optional `ChatHistory` can be provided to capture the complete messages for further analysis if needed. Since the assistant maintains the conversation state as a remote thread, capturing these messages is not always necessary. The decision to store and analyze the full response depends on the specific requirements of the interaction. +When invoking a streamed response from an [`OpenAIAssistantAgent`](./assistant-agent.md), the assistant maintains the conversation state as a remote thread. It is possible to read the messages from the remote thread if required. ::: zone pivot="programming-language-csharp" ```csharp @@ -114,37 +118,244 @@ When invoking a streamed response from an [`OpenAIAssistantAgent`](./assistant-a OpenAIAssistantAgent agent = ...; // Create a thread for the agent conversation. -string threadId = await agent.CreateThreadAsync(); +OpenAIAssistantAgentThread agentThread = new(assistantClient); + +// Cerate a user message +var message = new ChatMessageContent(AuthorRole.User, ""); + +// Generate the streamed agent response(s) +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) +{ + // Process streamed response(s)... +} + +// It's possible to read the messages from the remote thread. +await foreach (ChatMessageContent response in agentThread.GetMessagesAsync()) +{ + // Process messages... +} + +// Delete the thread when it is no longer needed +await agentThread.DeleteAsync(); +``` + +To create a thread using an existing `Id`, pass it to the constructor of `OpenAIAssistantAgentThread`: + +```csharp +// Define agent +OpenAIAssistantAgent agent = ...; + +// Create a thread for the agent conversation. +OpenAIAssistantAgentThread agentThread = new(assistantClient, "your-existing-thread-id"); -// Add a user message to the conversation -chat.Add(threadId, new ChatMessageContent(AuthorRole.User, "")); +// Cerate a user message +var message = new ChatMessageContent(AuthorRole.User, ""); // Generate the streamed agent response(s) -await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(threadId)) +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) { // Process streamed response(s)... } +// It's possible to read the messages from the remote thread. +await foreach (ChatMessageContent response in agentThread.GetMessagesAsync()) +{ + // Process messages... +} + // Delete the thread when it is no longer needed -await agent.DeleteThreadAsync(threadId); +await agentThread.DeleteAsync(); ``` ::: zone-end ::: zone pivot="programming-language-python" ```python +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent, OpenAIAssistantAgent + # Define agent -agent = OpenAIAssistantAgent(...) +agent = OpenAIAssistantAgent(...) # or = AzureAssistantAgent(...) # Create a thread for the agent conversation. -thread_id = await agent.create_thread() +# If no thread is provided one will be created and returned with +# the initial response. +thread: AssistantAgentThread = None + +# Generate the streamed agent response(s) +async for response in agent.invoke_stream(messages="user input", thread=thread): + # Process streamed response(s)... + thread = response.thread -# Add user message to the conversation -await agent.add_chat_message(message="") +# Read the messages from the remote thread +async for response in thread.get_messages(): + # Process messages + +# Delete the thread +await thread.delete() +``` + +To create a thread using an existing `thread_id`, pass it to the constructor of `AssistantAgentThread`: + +```python +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent, OpenAIAssistantAgent + +# Define agent +agent = OpenAIAssistantAgent(...) # or = AzureAssistantAgent(...) + +# Create a thread for the agent conversation. +# If no thread is provided one will be created and returned with +# the initial response. +thread = AssistantAgentThread(client=client, thread_id="your-existing-thread-id") # Generate the streamed agent response(s) -async for response in agent.invoke_stream(thread_id=thread_id): +async for response in agent.invoke_stream(messages="user input", thread=thread): # Process streamed response(s)... + thread = response.thread + +# Delete the thread +await thread.delete() +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Handling Intermediate Messages with a Streaming Response + +The nature of streaming responses allows LLM models to return incremental chunks of text, enabling quicker rendering in a UI or console without waiting for the entire response to complete. Additionally, a caller might want to handle intermediate content, such as results from function calls. This can be achieved by supplying a callback function when invoking the streaming response. The callback function receives complete messages encapsulated within `ChatMessageContent`. + +::: zone pivot="programming-language-csharp" +> Callback documentation for the `AzureAIAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Configuring the `on_intermediate_message` callback within `agent.invoke_stream(...)` allows the caller to receive intermediate messages generated during the process of formulating the agent's final response. + +```python +import asyncio +from typing import Annotated + +from semantic_kernel.agents import AzureResponsesAgent +from semantic_kernel.functions import kernel_function + + +# Define a sample plugin for the sample +class MenuPlugin: + """A sample Menu Plugin used for the concept sample.""" + + @kernel_function(description="Provides a list of specials from the menu.") + def get_specials(self, menu_item: str) -> Annotated[str, "Returns the specials from the menu."]: + return """ + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + """ + + @kernel_function(description="Provides the price of the requested menu item.") + def get_item_price( + self, menu_item: Annotated[str, "The name of the menu item."] + ) -> Annotated[str, "Returns the price of the menu item."]: + return "$9.99" + + +# Simulate a conversation with the agent +USER_INPUTS = [ + "Hello", + "What is the special soup?", + "What is the special drink?", + "How much is it?", + "Thank you", +] + + +async def main(): + # 1. Create the client using OpenAI resources and configuration + client, model = AzureResponsesAgent.setup_resources() + + # 2. Create a Semantic Kernel agent for the OpenAI Responses API + agent = AzureResponsesAgent( + ai_model_id=model, + client=client, + instructions="Answer questions about the menu.", + name="Host", + plugins=[MenuPlugin()], + ) + + # 3. Create a thread for the agent + # If no thread is provided, a new thread will be + # created and returned with the initial response + thread = None + + for user_input in USER_INPUTS: + print(f"# User: '{user_input}'") + first_chunk = True + # 4. Invoke the agent for the current message and print the response + async for response in agent.invoke_stream(messages=user_input, thread=thread): + thread = response.thread + if first_chunk: + print(f"# {response.name}: ", end="", flush=True) + first_chunk = False + print(response.content, end="", flush=True) + print() + + # Print the intermediate steps + print("\nIntermediate Steps:") + for msg in intermediate_steps: + if any(isinstance(item, FunctionResultContent) for item in msg.items): + for fr in msg.items: + if isinstance(fr, FunctionResultContent): + print(f"Function Result:> {fr.result} for function: {fr.name}") + elif any(isinstance(item, FunctionCallContent) for item in msg.items): + for fcc in msg.items: + if isinstance(fcc, FunctionCallContent): + print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}") + else: + print(f"{msg.role}: {msg.content}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + +The following demonstrates sample output from the agent invocation process: + +```bash +Sample Output: + +# AuthorRole.USER: 'Hello' +# Host: Hi there! How can I assist you with the menu today? +# AuthorRole.USER: 'What is the special soup?' +# Host: The special soup is Clam Chowder. +# AuthorRole.USER: 'What is the special drink?' +# Host: The special drink is Chai Tea. +# AuthorRole.USER: 'How much is that?' +# Host: Could you please specify the menu item you are asking about? +# AuthorRole.USER: 'Thank you' +# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. + +Intermediate Steps: +AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today? +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special soup is Clam Chowder. +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special drink is Chai Tea. +AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about? +AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. ``` + ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/Frameworks/agent/agent-templates.md b/semantic-kernel/Frameworks/agent/agent-templates.md index 6813d6f9..00148161 100644 --- a/semantic-kernel/Frameworks/agent/agent-templates.md +++ b/semantic-kernel/Frameworks/agent/agent-templates.md @@ -10,14 +10,11 @@ ms.service: semantic-kernel --- # Create an Agent from a Semantic Kernel Template -> [!IMPORTANT] -> This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. - ## Prompt Templates in Semantic Kernel An agent's role is primarily shaped by the instructions it receives, which dictate its behavior and actions. Similar to invoking a `Kernel` [prompt](../../concepts/prompts/index.md), an agent's instructions can include templated parameters—both values and functions—that are dynamically substituted during execution. This enables flexible, context-aware responses, allowing the agent to adjust its output based on real-time input. -Additionally, an agent can be configured directly using a _Prompt Template Configuration_, providing developers with a structured and reusable way to define its behavior. This approach offers a powerful tool for standardizing and customizing agent instructions, ensuring consistency across various use cases while still maintaining dynamic adaptability. +Additionally, an agent can be configured directly using a Prompt Template Configuration, providing developers with a structured and reusable way to define its behavior. This approach offers a powerful tool for standardizing and customizing agent instructions, ensuring consistency across various use cases while still maintaining dynamic adaptability. #### Related API's: @@ -76,10 +73,8 @@ ChatCompletionAgent agent = ::: zone pivot="programming-language-python" ```python -kernel = Kernel() - agent = ChatCompletionAgent( - kernel=kernel, + service=AzureChatCompletion(), # or other supported AI Services name="StoryTeller", instructions="Tell a story about {{$topic}} that is {{$length}} sentences long.", arguments=KernelArguments(topic="Dog", length="2"), @@ -100,24 +95,34 @@ Templated instructions are especially powerful when working with an [`OpenAIAssi ::: zone pivot="programming-language-csharp" ```csharp // Retrieve an existing assistant definition by identifier -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.RetrieveAsync( - this.GetClientProvider(), - "", - new Kernel(), - new KernelArguments() - { - { "topic", "Dog" }, - { "length", "3" }, - }); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri("")); +AssistantClient assistantClient = client.GetAssistantClient(); +Assistant assistant = await client.GetAssistantAsync(); +OpenAIAssistantAgent agent = new(assistant, assistantClient, new KernelPromptTemplateFactory(), PromptTemplateConfig.SemanticKernelTemplateFormat) +{ + Arguments = new KernelArguments() + { + { "topic", "Dog" }, + { "length", "3" }, + } +} ``` ::: zone-end ::: zone pivot="programming-language-python" ```python -agent = await OpenAIAssistantAgent.retrieve( - id=, - kernel=Kernel(), +# Create the client using Azure OpenAI resources and configuration +client, model = AzureAssistantAgent.setup_resources() + +# Retrieve the assistant definition from the server based on the assistant ID +definition = await client.beta.assistants.retrieve( + assistant_id="your-assistant-id", +) + +# Create the AzureAssistantAgent instance using the client and the assistant definition +agent = AzureAssistantAgent( + client=client, + definition=definition, arguments=KernelArguments(topic="Dog", length="3"), ) ``` @@ -130,10 +135,9 @@ agent = await OpenAIAssistantAgent.retrieve( ::: zone-end -## Agent Definition from a _Prompt Template_ - -The same _Prompt Template Config_ used to create a _Kernel Prompt Function_ can also be leveraged to define an agent. This allows for a unified approach in managing both prompts and agents, promoting consistency and reuse across different components. By externalizing agent definitions from the codebase, this method simplifies the management of multiple agents, making them easier to update and maintain without requiring changes to the underlying logic. This separation also enhances flexibility, enabling developers to modify agent behavior or introduce new agents by simply updating the configuration, rather than adjusting the code itself. +## Agent Definition from a Prompt Template +The same Prompt Template Config used to create a Kernel Prompt Function can also be leveraged to define an agent. This allows for a unified approach in managing both prompts and agents, promoting consistency and reuse across different components. By externalizing agent definitions from the codebase, this method simplifies the management of multiple agents, making them easier to update and maintain without requiring changes to the underlying logic. This separation also enhances flexibility, enabling developers to modify agent behavior or introduce new agents by simply updating the configuration, rather than adjusting the code itself. #### YAML Template ```yaml @@ -192,7 +196,7 @@ data = yaml.safe_load(generate_story_yaml) prompt_template_config = PromptTemplateConfig(**data) agent = ChatCompletionAgent( - kernel=_create_kernel_with_chat_completion(), + service=AzureChatCompletion(), # or other supported AI services prompt_template_config=prompt_template_config, arguments=KernelArguments(topic="Dog", length="3"), ) @@ -228,9 +232,6 @@ ChatCompletionAgent agent = } }; -// Create a ChatHistory object to maintain the conversation state. -ChatHistory chat = []; - KernelArguments overrideArguments = new() { @@ -239,7 +240,7 @@ KernelArguments overrideArguments = }); // Generate the agent response(s) -await foreach (ChatMessageContent response in agent.InvokeAsync(chat, overrideArguments)) +await foreach (ChatMessageContent response in agent.InvokeAsync([], options: new() { KernelArguments = overrideArguments })) { // Process agent response(s)... } @@ -249,28 +250,30 @@ await foreach (ChatMessageContent response in agent.InvokeAsync(chat, overrideAr ::: zone pivot="programming-language-python" ```python -kernel = Kernel() - agent = ChatCompletionAgent( - kernel=kernel, + service=AzureChatCompletion(), name="StoryTeller", instructions="Tell a story about {{$topic}} that is {{$length}} sentences long.", arguments=KernelArguments(topic="Dog", length="2"), ) -# Create a chat history to maintain the conversation state -chat = ChatHistory() +# Create a thread to maintain the conversation state +# If no threaded is created, a thread will be returned +# with the initial response +thread = None override_arguments = KernelArguments(topic="Cat", length="3") # Two ways to get a response from the agent # Get the response which returns a ChatMessageContent directly -response = await agent.get_response(chat, arguments=override_arguments) +response = await agent.get_response(messages="user input", arguments=override_arguments) +thread = response.thread # or use the invoke method to return an AsyncIterable of ChatMessageContent -async for response in agent.invoke(chat, arguments=override_arguments): +async for response in agent.invoke(messages="user input", arguments=override_arguments): # process agent response(s)... + thread = response.thread ``` ::: zone-end diff --git a/semantic-kernel/Frameworks/agent/assistant-agent.md b/semantic-kernel/Frameworks/agent/assistant-agent.md index 17ee3199..1e70b10e 100644 --- a/semantic-kernel/Frameworks/agent/assistant-agent.md +++ b/semantic-kernel/Frameworks/agent/assistant-agent.md @@ -8,24 +8,19 @@ ms.author: crickman ms.date: 09/13/2024 ms.service: semantic-kernel --- -# Exploring the _Semantic Kernel_ `OpenAIAssistantAgent` - -> [!IMPORTANT] -> This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. +# Exploring the Semantic Kernel `OpenAIAssistantAgent` Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-csharp" - [`OpenAIAssistantAgent`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiassistantagent) -- [`OpenAIAssistantDefinition`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiassistantdefinition) -- [`OpenAIClientProvider`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiclientprovider) ::: zone-end ::: zone pivot="programming-language-python" -- [`azure_assistant_agent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.azure_assistant_agent) -- [`open_ai_assistant_agent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.open_ai_assistant_agent) +- [`AzureAssistantAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.azure_assistant_agent.azureassistantagent) +- [`OpenAIAssistantAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.open_ai_assistant_agent.openaiassistantagent) ::: zone-end @@ -38,34 +33,68 @@ Detailed API documentation related to this discussion is available at: ## What is an Assistant? -The _OpenAI Assistant API_ is a specialized interface designed for more advanced and interactive AI capabilities, enabling developers to create personalized and multi-step task-oriented agents. Unlike the Chat Completion API, which focuses on simple conversational exchanges, the Assistant API allows for dynamic, goal-driven interactions with additional features like code-interpreter and file-search. +The OpenAI Assistants API is a specialized interface designed for more advanced and interactive AI capabilities, enabling developers to create personalized and multi-step task-oriented agents. Unlike the Chat Completion API, which focuses on simple conversational exchanges, the Assistant API allows for dynamic, goal-driven interactions with additional features like code-interpreter and file-search. - [OpenAI Assistant Guide](https://platform.openai.com/docs/assistants) - [OpenAI Assistant API](https://platform.openai.com/docs/api-reference/assistants) - [Assistant API in Azure](/azure/ai-services/openai/assistants-quickstart) +## Preparing Your Development Environment + +To proceed with developing an `OpenAIAIAssistantAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" + +Add the `Microsoft.SemanticKernel.Agents.OpenAI` package to your project: + +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.AzureAI --prerelease +``` + +You may also want to include the `Azure.Identity` package: + +```pwsh +dotnet add package Azure.Identity +``` +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package with the optional Azure dependencies: + +```bash +pip install semantic-kernel[azure] +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + ## Creating an `OpenAIAssistantAgent` -Creating an `OpenAIAssistant` requires invoking a remote service, which is handled asynchronously. To manage this, the `OpenAIAssistantAgent` is instantiated through a static factory method, ensuring the process occurs in a non-blocking manner. This method abstracts the complexity of the asynchronous call, returning a promise or future once the assistant is fully initialized and ready for use. +Creating an `OpenAIAssistant` requires first creating a client to be able to talk a remote service. ::: zone pivot="programming-language-csharp" ```csharp -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - OpenAIClientProvider.ForAzureOpenAI(/*<...service configuration>*/), - new OpenAIAssistantDefinition("") - { - Name = "", - Instructions = "", - }, - new Kernel()); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +Assistant assistant = + await client.CreateAssistantAsync( + "", + "", + instructions: ""); +OpenAIAssistantAgent agent = new(assistant, client); ``` ::: zone-end ::: zone pivot="programming-language-python" ```python -from semantic_kernel.agents.open_ai import AzureAssistantAgent, OpenAIAssistantAgent +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent, OpenAIAssistantAgent # Set up the client and model using Azure OpenAI Resources client, model = AzureAssistantAgent.setup_resources() @@ -119,11 +148,9 @@ Once created, the identifier of the assistant may be access via its identifier. For .NET, the agent identifier is exposed as a `string` via the property defined by any agent. ```csharp -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.RetrieveAsync( - OpenAIClientProvider.ForAzureOpenAI(/*<...service configuration>*/), - "", - new Kernel()); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +Assistant assistant = await client.GetAssistantAsync(""); +OpenAIAssistantAgent agent = new(assistant, client); ``` ::: zone-end @@ -164,50 +191,78 @@ agent = AzureAssistantAgent( ## Using an `OpenAIAssistantAgent` -As with all aspects of the _Assistant API_, conversations are stored remotely. Each conversation is referred to as a _thread_ and identified by a unique `string` identifier. Interactions with your `OpenAIAssistantAgent` are tied to this specific thread identifier which must be specified when calling the agent/ +As with all aspects of the Assistant API, conversations are stored remotely. Each conversation is referred to as a thread and identified by a unique `string` identifier. Interactions with your `OpenAIAssistantAgent` are tied to this specific thread identifier. The specifics of the Assistant API thread is abstracted away via the `OpenAIAssistantAgentThread` class, which is an implementation of `AgentThread`. + +The `OpenAIAssistantAgent` currently only supports threads of type `OpenAIAssistantAgentThread`. + +You can invoke the `OpenAIAssistantAgent` without specifying an `AgentThread`, to start a new thread and a new `AgentThread` will be returned as part of the response. ::: zone pivot="programming-language-csharp" ```csharp + // Define agent OpenAIAssistantAgent agent = ...; +AgentThread? agentThread = null; + +// Generate the agent response(s) +await foreach (AgentResponseItem response in agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, ""))) +{ + // Process agent response(s)... + agentThread = response.Thread; +} + +// Delete the thread if no longer needed +if (agentThread is not null) +{ + await agentThread.DeleteAsync(); +} +``` -// Create a thread for the agent conversation. -string threadId = await agent.CreateThreadAsync(); +You can also invoke the `OpenAIAssistantAgent` with an `AgentThread` that you created. -// Add a user message to the conversation -chat.Add(threadId, new ChatMessageContent(AuthorRole.User, "")); +```csharp +// Define agent +OpenAIAssistantAgent agent = ...; + +// Create a thread with some custom metadata. +AgentThread agentThread = new OpenAIAssistantAgentThread(client, metadata: myMetadata); // Generate the agent response(s) -await foreach (ChatMessageContent response in agent.InvokeAsync(threadId)) +await foreach (ChatMessageContent response in agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, ""), agentThread)) { // Process agent response(s)... } // Delete the thread when it is no longer needed -await agent.DeleteThreadAsync(threadId); +await agentThread.DeleteAsync(); +``` + +You can also create an `OpenAIAssistantAgentThread` that resumes an earlier conversation by id. + +```csharp +// Create a thread with an existing thread id. +AgentThread agentThread = new OpenAIAssistantAgentThread(client, "existing-thread-id"); ``` + ::: zone-end ::: zone pivot="programming-language-python" ```python +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent + # Define agent openai_agent = await ... # Create a thread for the agent conversation -thread_id = await agent.create_thread() - -# Add a user message to the conversation -await agent.add_chat_message( - thread_id=thread_id, - message=ChatMessageContent(role=AuthorRole.USER, content=""), -) +thread: AssistantAgentThread = None # Generate the agent response(s) -async for response in agent.invoke(thread_id=thread_id): +async for response in agent.invoke(messages="user input", thread=thread): # process agent response(s)... + thread = response.thread # Delete the thread when it is no longer needed -await agent.delete_thread(thread_id) +await thread.delete() if thread else None ``` ::: zone-end @@ -220,20 +275,18 @@ await agent.delete_thread(thread_id) ## Deleting an `OpenAIAssistantAgent` -Since the assistant's definition is stored remotely, it supports the capability to self-delete. This enables the agent to be removed from the system when it is no longer needed. +Since the assistant's definition is stored remotely, it will persist if not deleted. +Deleting an assistant definition may be performed directly with the `AssistantClient`. -> Note: Attempting to use an agent instance after being deleted results in an exception. +> Note: Attempting to use an agent instance after being deleted will result in a service exception. ::: zone pivot="programming-language-csharp" For .NET, the agent identifier is exposed as a `string` via the [`Agent.Id`](/dotnet/api/microsoft.semantickernel.agents.agent.id) property defined by any agent. ```csharp -// Perform the deletion -await agent.DeleteAsync(); - -// Inspect whether an agent has been deleted -bool isDeleted = agent.IsDeleted(); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +await client.DeleteAssistantAsync(""); ``` ::: zone-end @@ -251,6 +304,154 @@ is_deleted = agent._is_deleted ::: zone-end +## Handling Intermediate Messages with an `OpenAIAssistantAgent` + +The Semantic Kernel `OpenAIAssistantAgent` is designed to invoke an agent that fulfills user queries or questions. During invocation, the agent may execute tools to derive the final answer. To access intermediate messages produced during this process, callers can supply a callback function that handles instances of `FunctionCallContent` or `FunctionResultContent`. + +::: zone pivot="programming-language-csharp" +> Callback documentation for the `OpenAIAssistantAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Configuring the `on_intermediate_message` callback within `agent.invoke(...)` or `agent.invoke_stream(...)` allows the caller to receive intermediate messages generated during the process of formulating the agent's final response. + +```python +import asyncio +from typing import Annotated + +from semantic_kernel.agents import AzureAssistantAgent +from semantic_kernel.contents import AuthorRole, ChatMessageContent, FunctionCallContent, FunctionResultContent +from semantic_kernel.functions import kernel_function + +class MenuPlugin: + """A sample Menu Plugin used for the concept sample.""" + + @kernel_function(description="Provides a list of specials from the menu.") + def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]: + return """ + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + """ + + @kernel_function(description="Provides the price of the requested menu item.") + def get_item_price( + self, menu_item: Annotated[str, "The name of the menu item."] + ) -> Annotated[str, "Returns the price of the menu item."]: + return "$9.99" + +# Define a list to hold callback message content +intermediate_steps: list[ChatMessageContent] = [] + +# Define an async method to handle the `on_intermediate_message` callback +async def handle_intermediate_steps(message: ChatMessageContent) -> None: + intermediate_steps.append(message) + +async def main(): + # Set up the client and model using Azure OpenAI Resources + client, model = AzureAssistantAgent.setup_resources() + + # Define the assistant definition + definition = await client.beta.assistants.create( + model=model, + instructions="", + name="", + ) + + # Create the AzureAssistantAgent instance using the client and the assistant definition + agent = AzureAssistantAgent( + client=client, + definition=definition, + plugins=[MenuPlugin()] + ) + + user_inputs = [ + "Hello", + "What is the special soup?", + "What is the special drink?", + "How much is that?", + "Thank you", + ] + + thread = None + + # Generate the agent response(s) + for user_input in user_inputs: + print(f"# {AuthorRole.USER}: '{user_input}'") + async for response in agent.invoke( + messages=user_input, + thread=thread, + on_intermediate_message=handle_intermediate_steps, + ): + thread = response.thread + print(f"# {response.name}: {response.content}") + + # Delete the thread when it is no longer needed + await thread.delete() if thread else None + + # Print the intermediate steps + print("\nIntermediate Steps:") + for msg in intermediate_steps: + if any(isinstance(item, FunctionResultContent) for item in msg.items): + for fr in msg.items: + if isinstance(fr, FunctionResultContent): + print(f"Function Result:> {fr.result} for function: {fr.name}") + elif any(isinstance(item, FunctionCallContent) for item in msg.items): + for fcc in msg.items: + if isinstance(fcc, FunctionCallContent): + print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}") + else: + print(f"{msg.role}: {msg.content}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + +The following demonstrates sample output from the agent invocation process: + +```bash +Sample Output: + +# AuthorRole.USER: 'Hello' +# Host: Hi there! How can I assist you with the menu today? +# AuthorRole.USER: 'What is the special soup?' +# Host: The special soup is Clam Chowder. +# AuthorRole.USER: 'What is the special drink?' +# Host: The special drink is Chai Tea. +# AuthorRole.USER: 'How much is that?' +# Host: Could you please specify the menu item you are asking about? +# AuthorRole.USER: 'Thank you' +# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. + +Intermediate Steps: +AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today? +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special soup is Clam Chowder. +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special drink is Chai Tea. +AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about? +AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + ## How-To @@ -261,5 +462,5 @@ For an end-to-end example for a `OpenAIAssistantAgent`, see: > [!div class="nextstepaction"] -> [Agent Collaboration in `AgentChat`](./agent-chat.md) +> [Explore the Azure AI Agent](./azure-ai-agent.md) diff --git a/semantic-kernel/Frameworks/agent/azure-ai-agent.md b/semantic-kernel/Frameworks/agent/azure-ai-agent.md new file mode 100644 index 00000000..de739ba3 --- /dev/null +++ b/semantic-kernel/Frameworks/agent/azure-ai-agent.md @@ -0,0 +1,852 @@ +--- +title: Exploring the Semantic Kernel Azure AI Agent Agent +description: An exploration of the definition, behaviors, and usage patterns for an Azure AI Agent +zone_pivot_groups: programming-languages +author: moonbox3 +ms.topic: tutorial +ms.author: evmattso +ms.date: 03/05/2025 +ms.service: semantic-kernel +--- +# Exploring the Semantic Kernel `AzureAIAgent` + +> [!IMPORTANT] +> This feature is in the experimental stage. Features at this stage are under development and subject to change before advancing to the preview or release candidate stage. + +Detailed API documentation related to this discussion is available at: + +::: zone pivot="programming-language-csharp" + +- [`AzureAIAgent`](/dotnet/api/microsoft.semantickernel.agents.azureai) + +::: zone-end + +::: zone pivot="programming-language-python" + +- [`AzureAIAgent`](/python/api/semantic-kernel/semantic_kernel.agents.azure_ai.azure_ai_agent.azureaiagent) + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## What is an `AzureAIAgent`? + +An `AzureAIAgent` is a specialized agent within the Semantic Kernel framework, designed to provide advanced conversational capabilities with seamless tool integration. It automates tool calling, eliminating the need for manual parsing and invocation. The agent also securely manages conversation history using threads, reducing the overhead of maintaining state. Additionally, the `AzureAIAgent` supports a variety of built-in tools, including file retrieval, code execution, and data interaction via Bing, Azure AI Search, Azure Functions, and OpenAPI. + +To use an `AzureAIAgent`, an Azure AI Foundry Project must be utilized. The following articles provide an overview of the Azure AI Foundry, how to create and configure a project, and the agent service: + +- [What is Azure AI Foundry?](/azure/ai-foundry/what-is-ai-foundry) +- [The Azure AI Foundry SDK](/azure/ai-foundry/how-to/develop/sdk-overview) +- [What is Azure AI Agent Service](/azure/ai-services/agents/overview) +- [Quickstart: Create a new agent](/azure/ai-services/agents/quickstart) + + +## Preparing Your Development Environment + +To proceed with developing an `AzureAIAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" + +Add the `Microsoft.SemanticKernel.Agents.AzureAI` package to your project: + +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.AzureAI --prerelease +``` + +You may also want to include the `Azure.Identity` package: + +```pwsh +dotnet add package Azure.Identity +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package with the optional Azure dependencies: + +```bash +pip install semantic-kernel[azure] +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + +## Configuring the AI Project Client + +Accessing an `AzureAIAgent` first requires the creation of a project client that is configured for a specific Foundry Project, most commonly by providing a connection string ([The Azure AI Foundry SDK: Getting Started with Projects](/azure/ai-foundry/how-to/develop/sdk-overview#get-started-with-projects)). + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +``` + +The `AgentsClient` may be accessed from the `AIProjectClient`: + +```c# +AgentsClient agentsClient = client.GetAgentsClient(); +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +Modify your the `.env` file in the root directory to include: + +```bash +AZURE_AI_AGENT_PROJECT_CONNECTION_STRING = "" +AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "" +``` + +or + +```bash +AZURE_AI_AGENT_ENDPOINT = "" +AZURE_AI_AGENT_SUBSCRIPTION_ID = "" +AZURE_AI_AGENT_RESOURCE_GROUP_NAME = "" +AZURE_AI_AGENT_PROJECT_NAME = "" +AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "" +``` + +Once the configuration is defined, the client may be created: + +```python +from semantic_kernel.agents import AzureAIAgent + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, +): + # Your operational code here +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Creating an `AzureAIAgent` + +To create an `AzureAIAgent`, you start by configuring and initializing the agent project through the Azure AI service and then integrate it with Semantic Kernel: + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +// 1. Define an agent on the Azure AI agent service +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: ""); + +// 2. Create a Semantic Kernel agent based on the agent definition +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" + +```python +from azure.identity.aio import DefaultAzureCredential +from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings + +ai_agent_settings = AzureAIAgentSettings.create() + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, +): + # 1. Define an agent on the Azure AI agent service + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + name="", + instructions="", + ) + + # 2. Create a Semantic Kernel agent based on the agent definition + agent = AzureAIAgent( + client=client, + definition=agent_definition, + ) +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Interacting with an `AzureAIAgent` + +Interaction with the `AzureAIAgent` is straightforward. The agent maintains the conversation history automatically using a thread. +::: zone pivot="programming-language-csharp" +The specifics of the _Azure AI Agent thread_ is abstracted away via the `Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgentThread` class, which is an implementation of `Microsoft.SemanticKernel.Agents.AgentThread`. + +> [!IMPORTANT] +> Note that the Azure AI Agents SDK has the `Azure.AI.Projects.AgentThread` class. It should not be confused with `Microsoft.SemanticKernel.Agents.AgentThread`, which is the common Semantic Kernel Agents abstraction for all thread types. + +The `AzureAIAgent` currently only supports threads of type `AzureAIAgentThread`. + +```c# +Microsoft.SemanticKernel.Agents.AgentThread agentThread = new AzureAIAgentThread(agent.Client); +try +{ + ChatMessageContent message = new(AuthorRole.User, ""); + await foreach (ChatMessageContent response in agent.InvokeAsync(message, agentThread)) + { + Console.WriteLine(response.Content); + } +} +finally +{ + await agentThread.DeleteAsync(); + await agent.Client.DeleteAgentAsync(agent.Id); +} +``` + +::: zone-end + +::: zone pivot="programming-language-python" +The specifics of the _Azure AI Agent thread_ is abstracted away via the `AzureAIAgentThread` class, which is an implementation of `AgentThread`. + +```python +USER_INPUTS = ["Hello", "What's your name?"] + +thread: AzureAIAgentThread = AzureAIAgentThread() + +try: + for user_input in USER_INPUTS: + response = await agent.get_response(messages=user_inputs, thread=thread) + print(response) + thread = response.thread +finally: + await thread.delete() if thread else None +``` + +Optionally, an agent may be invoked as: + +```python +for user_input in USER_INPUTS: + async for content in agent.invoke(messages=user_input, thread=thread): + print(content.content) + thread = response.thread +``` + +You may also pass in a list of messages to the `get_response(...)`, `invoke(...)`, or `invoke_stream(...)` methods: + +```python +USER_INPUTS = ["Hello", "What's your name?"] + +thread: AzureAIAgentThread = AzureAIAgentThread() + +try: + for user_input in USER_INPUTS: + response = await agent.get_response(messages=USER_INPUTS, thread=thread) + print(response) + thread = response.thread +finally: + await thread.delete() if thread else None +``` + +::: zone-end + +An agent may also produce a streamed response: + +::: zone pivot="programming-language-csharp" +```c# +ChatMessageContent message = new(AuthorRole.User, ""); +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) +{ + Console.Write(response.Content); +} +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +for user_input in USER_INPUTS: + await agent.add_chat_message(thread_id=thread.id, message=user_input) + async for content in agent.invoke_stream(thread_id=thread.id): + print(content.content, end="", flush=True) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Using Plugins with an `AzureAIAgent` + +Semantic Kernel supports extending an `AzureAIAgent` with custom plugins for enhanced functionality: + +::: zone pivot="programming-language-csharp" +```c# +KernelPlugin plugin = KernelPluginFactory.CreateFromType(); +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: ""); + +AzureAIAgent agent = new(definition, agentsClient, plugins: [plugin]); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from semantic_kernel.functions import kernel_function + +class SamplePlugin: + @kernel_function(description="Provides sample data.") + def get_data(self) -> str: + return "Sample data" + +ai_agent_settings = AzureAIAgentSettings.create() + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + ) + + agent = AzureAIAgent( + client=client, + definition=agent_definition, + plugins=[SamplePlugin()] + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Advanced Features + +An `AzureAIAgent` can leverage advanced tools such as: + +- [Code Interpreter](#code-interpreter) +- [File Search](#file-search) +- [OpenAPI integration](#openapi-integration) +- [Azure AI Search integration](#azureai-search-integration) + +### Code Interpreter + +Code Interpreter allows the agents to write and run Python code in a sandboxed execution environment ([Azure AI Agent Service Code Interpreter](/azure/ai-services/agents/how-to/tools/code-interpreter)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new CodeInterpreterToolDefinition()], + toolResources: + new() + { + CodeInterpreter = new() + { + FileIds = { ... }, + } + })); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import CodeInterpreterTool + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + code_interpreter = CodeInterpreterTool() + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=code_interpreter.definitions, + tool_resources=code_interpreter.resources, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### File Search + +File search augments agents with knowledge from outside its model ([Azure AI Agent Service File Search Tool](/azure/ai-services/agents/how-to/tools/file-search)). + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new FileSearchToolDefinition()], + toolResources: + new() + { + FileSearch = new() + { + VectorStoreIds = { ... }, + } + }); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import FileSearchTool + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + file_search = FileSearchTool(vector_store_ids=[vector_store.id]) + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=file_search.definitions, + tool_resources=file_search.resources, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### OpenAPI Integration + +Connects your agent to an external API ([How to use Azure AI Agent Service with OpenAPI Specified Tools](/azure/ai-services/agents/how-to/tools/openapi-spec)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +string apiJsonSpecification = ...; // An Open API JSON specification + +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [ + new OpenApiToolDefinition( + "", + "", + BinaryData.FromString(apiJsonSpecification), + new OpenApiAnonymousAuthDetails()) + ] +); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import OpenApiTool, OpenApiAnonymousAuthDetails + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + openapi_spec_file_path = "sample/filepath/..." + with open(os.path.join(openapi_spec_file_path, "spec_one.json")) as file_one: + openapi_spec_one = json.loads(file_one.read()) + with open(os.path.join(openapi_spec_file_path, "spec_two.json")) as file_two: + openapi_spec_two = json.loads(file_two.read()) + + # Note that connection or managed identity auth setup requires additional setup in Azure + auth = OpenApiAnonymousAuthDetails() + openapi_tool_one = OpenApiTool( + name="", + spec=openapi_spec_one, + description="", + auth=auth, + ) + openapi_tool_two = OpenApiTool( + name="", + spec=openapi_spec_two, + description="", + auth=auth, + ) + + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=openapi_tool_one.definitions + openapi_tool_two.definitions, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### AzureAI Search Integration + +Use an existing Azure AI Search index with with your agent ([Use an existing AI Search index](/azure/ai-services/agents/how-to/tools/azure-ai-search)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +ConnectionsClient cxnClient = client.GetConnectionsClient(); +ListConnectionsResponse searchConnections = await cxnClient.GetConnectionsAsync(Azure.AI.Projects.ConnectionType.AzureAISearch); +ConnectionResponse searchConnection = searchConnections.Value[0]; + +Azure.AI.Projects.Agent definition = await agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new Azure.AI.Projects.AzureAISearchToolDefinition()], + toolResources: new() + { + AzureAISearch = new() + { + IndexList = { new Azure.AI.Projects.IndexResource(searchConnection.Id, "") } + } + }); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import AzureAISearchTool, ConnectionType + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + conn_list = await client.connections.list() + + ai_search_conn_id = "" + for conn in conn_list: + if conn.connection_type == ConnectionType.AZURE_AI_SEARCH: + ai_search_conn_id = conn.id + break + + ai_search = AzureAISearchTool( + index_connection_id=ai_search_conn_id, + index_name=AZURE_AI_SEARCH_INDEX_NAME, + ) + + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + instructions="Answer questions using your index.", + tools=ai_search.definitions, + tool_resources=ai_search.resources, + headers={"x-ms-enable-preview": "true"}, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### Retrieving an Existing `AzureAIAgent` + +An existing agent can be retrieved and reused by specifying its assistant ID: + +::: zone pivot="programming-language-csharp" + +```c# +Azure.AI.Projects.Agent definition = await agentsClient.GetAgentAsync(""); +AzureAIAgent agent = new(definition, agentsClient); +``` + +::: zone-end + +::: zone pivot="programming-language-python" +```python +agent_definition = await client.agents.get_agent(assistant_id="your-agent-id") +agent = AzureAIAgent(client=client, definition=agent_definition) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Deleting an `AzureAIAgent` + +Agents and their associated threads can be deleted when no longer needed: + +::: zone pivot="programming-language-csharp" + +```c# +await agentThread.DeleteAsync(); +await agentsClient.DeleteAgentAsync(agent.Id); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +await client.agents.delete_thread(thread.id) +await client.agents.delete_agent(agent.id) +``` +::: zone-end + +If working with a vector store or files, they may be deleted as well: + +::: zone pivot="programming-language-csharp" +```c# +await agentsClient.DeleteVectorStoreAsync(""); +await agentsClient.DeleteFileAsync(""); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +await client.agents.delete_file(file_id=file.id) +await client.agents.delete_vector_store(vector_store_id=vector_store.id) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +> More information on the _file search_ tool is described in the [Azure AI Agent Service file search tool](/azure/ai-services/agents/how-to/tools/file-search) article. + +## How-To + +For practical examples of using an `AzureAIAgent`, see our code samples on GitHub: + +::: zone pivot="programming-language-csharp" + +- [Getting Started with Azure AI Agents](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/GettingStartedWithAgents/AzureAIAgent) +- [Advanced Azure AI Agent Code Samples](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/Concepts/Agents) + +::: zone-end + +::: zone pivot="programming-language-python" + +- [Getting Started with Azure AI Agents](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/getting_started_with_agents/azure_ai_agent) +- [Advanced Azure AI Agent Code Samples](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/agents/azure_ai_agent) + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Handling Intermediate Messages with an `AzureAIAgent` + +The Semantic Kernel `AzureAIAgent` is designed to invoke an agent that fulfills user queries or questions. During invocation, the agent may execute tools to derive the final answer. To access intermediate messages produced during this process, callers can supply a callback function that handles instances of `FunctionCallContent` or `FunctionResultContent`. + +::: zone pivot="programming-language-csharp" +> Callback documentation for the `AzureAIAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Configuring the `on_intermediate_message` callback within `agent.invoke(...)` or `agent.invoke_stream(...)` allows the caller to receive intermediate messages generated during the process of formulating the agent's final response. + +```python +import asyncio +from typing import Annotated + +from azure.identity.aio import DefaultAzureCredential + +from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings +from semantic_kernel.contents import ChatMessageContent, FunctionCallContent, FunctionResultContent +from semantic_kernel.functions import kernel_function + +class MenuPlugin: + """A sample Menu Plugin used for the concept sample.""" + + @kernel_function(description="Provides a list of specials from the menu.") + def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]: + return """ + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + """ + + @kernel_function(description="Provides the price of the requested menu item.") + def get_item_price( + self, menu_item: Annotated[str, "The name of the menu item."] + ) -> Annotated[str, "Returns the price of the menu item."]: + return "$9.99" + +# Define a list to hold callback message content +intermediate_steps: list[ChatMessageContent] = [] + +# Define an async method to handle the `on_intermediate_message` callback +async def handle_intermediate_steps(message: ChatMessageContent) -> None: + intermediate_steps.append(message) + +async def main(): + ai_agent_settings = AzureAIAgentSettings.create() + + async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client( + credential=creds, + conn_str=ai_agent_settings.project_connection_string.get_secret_value(), + ) as client, + ): + # Create agent definition + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + name="", + instructions="", + ) + + # Create the AzureAI Agent + agent = AzureAIAgent( + client=client, + definition=agent_definition, + plugins=[MenuPlugin()], + ) + + user_inputs = [ + "Hello", + "What is the special soup?", + "What is the special drink?", + "How much is that?", + "Thank you", + ] + + thread = None + + # Generate the agent response(s) + for user_input in user_inputs: + print(f"# {AuthorRole.USER}: '{user_input}'") + async for response in agent.invoke( + messages=user_input, + thread=thread, + on_intermediate_message=handle_intermediate_steps, + ): + thread = response.thread + print(f"# {response.name}: {response.content}") + + # Delete the thread when it is no longer needed + await thread.delete() if thread else None + + # Print the intermediate steps + print("\nIntermediate Steps:") + for msg in intermediate_steps: + if any(isinstance(item, FunctionResultContent) for item in msg.items): + for fr in msg.items: + if isinstance(fr, FunctionResultContent): + print(f"Function Result:> {fr.result} for function: {fr.name}") + elif any(isinstance(item, FunctionCallContent) for item in msg.items): + for fcc in msg.items: + if isinstance(fcc, FunctionCallContent): + print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}") + else: + print(f"{msg.role}: {msg.content}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + +The following demonstrates sample output from the agent invocation process: + +```bash +Sample Output: + +# AuthorRole.USER: 'Hello' +# Host: Hi there! How can I assist you with the menu today? +# AuthorRole.USER: 'What is the special soup?' +# Host: The special soup is Clam Chowder. +# AuthorRole.USER: 'What is the special drink?' +# Host: The special drink is Chai Tea. +# AuthorRole.USER: 'How much is that?' +# Host: Could you please specify the menu item you are asking about? +# AuthorRole.USER: 'Thank you' +# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. + +Intermediate Steps: +AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today? +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special soup is Clam Chowder. +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special drink is Chai Tea. +AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about? +AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +> [!div class="nextstepaction"] +> [Explore the OpenAI Responses Agent](./responses-agent.md) \ No newline at end of file diff --git a/semantic-kernel/Frameworks/agent/chat-completion-agent.md b/semantic-kernel/Frameworks/agent/chat-completion-agent.md index 0257948f..16bb3716 100644 --- a/semantic-kernel/Frameworks/agent/chat-completion-agent.md +++ b/semantic-kernel/Frameworks/agent/chat-completion-agent.md @@ -1,5 +1,5 @@ --- -title: Exploring the Semantic Kernel Chat Completion Agent +title: Exploring the Semantic Kernel ChatCompletionAgent description: An exploration of the definition, behaviors, and usage patterns for a Chat Completion Agent zone_pivot_groups: programming-languages author: crickman @@ -8,14 +8,12 @@ ms.author: crickman ms.date: 09/13/2024 ms.service: semantic-kernel --- -# Exploring the _Semantic Kernel_ Chat Completion Agent - -> [!IMPORTANT] -> This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. +# Exploring the Semantic Kernel `ChatCompletionAgent` Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-csharp" + - [`ChatCompletionAgent`](/dotnet/api/microsoft.semantickernel.agents.chatcompletionagent) - [`Microsoft.SemanticKernel.Agents`](/dotnet/api/microsoft.semantickernel.agents) - [`IChatCompletionService`](/dotnet/api/microsoft.semantickernel.chatcompletion.ichatcompletionservice) @@ -25,8 +23,7 @@ Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-python" -- [`chat_completion_agent`](/python/api/semantic-kernel/semantic_kernel.agents.chat_completion.chat_completion_agent) -- [`chat_completion_client_base`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.chat_completion_client_base) +- [`ChatCompletionAgent`](/python/api/semantic-kernel/semantic_kernel.agents.chat_completion.chat_completion_agent.chatcompletionagent) ::: zone-end @@ -36,34 +33,37 @@ Detailed API documentation related to this discussion is available at: ::: zone-end +## Chat Completion in Semantic Kernel -## Chat Completion in _Semantic Kernel_ +[Chat Completion](../../concepts/ai-services/chat-completion/index.md) is fundamentally a protocol for a chat-based interaction with an AI model where the chat-history is maintained and presented to the model with each request. Semantic Kernel [AI services](../../concepts/ai-services/index.md) offer a unified framework for integrating the chat-completion capabilities of various AI models. -[_Chat Completion_](../../concepts/ai-services/chat-completion/index.md) is fundamentally a protocol for a chat-based interaction with an AI model where the chat-history maintained and presented to the model with each request. _Semantic Kernel_ [AI services](../../concepts/ai-services/index.md) offer a unified framework for integrating the chat-completion capabilities of various AI models. +A `ChatCompletionAgent` can leverage any of these [AI services](../../concepts/ai-services/chat-completion/index.md) to generate responses, whether directed to a user or another agent. -A _chat completion agent_ can leverage any of these [AI services](../../concepts/ai-services/index.md) to generate responses, whether directed to a user or another agent. +## Preparing Your Development Environment -::: zone pivot="programming-language-csharp" +To proceed with developing an `ChatCompletionAgent`, configure your development environment with the appropriate packages. -For .NET, _chat-completion_ AI Services are based on the [`IChatCompletionService`](/dotnet/api/microsoft.semantickernel.chatcompletion.ichatcompletionservice) interface. +::: zone pivot="programming-language-csharp" -For .NET, some of AI services that support models with chat-completion include: +Add the `Microsoft.SemanticKernel.Agents.Core` package to your project: -Model|_Semantic Kernel_ AI Service ---|-- -Azure OpenAI|[`Microsoft.SemanticKernel.Connectors.AzureOpenAI`](/dotnet/api/microsoft.semantickernel.connectors.azureopenai) -Gemini|[`Microsoft.SemanticKernel.Connectors.Google`](/dotnet/api/microsoft.semantickernel.connectors.google) -HuggingFace|[`Microsoft.SemanticKernel.Connectors.HuggingFace`](/dotnet/api/microsoft.semantickernel.connectors.huggingface) -Mistral|[`Microsoft.SemanticKernel.Connectors.MistralAI`](/dotnet/api/microsoft.semantickernel.connectors.mistralai) -OpenAI|[`Microsoft.SemanticKernel.Connectors.OpenAI`](/dotnet/api/microsoft.semantickernel.connectors.openai) -Onnx|[`Microsoft.SemanticKernel.Connectors.Onnx`](/dotnet/api/microsoft.semantickernel.connectors.onnx) +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.Core --prerelease +``` ::: zone-end ::: zone pivot="programming-language-python" -- [`azure_chat_completion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion) -- [`open_ai_chat_completion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.open_ai_chat_completion) +Install the `semantic-kernel` package: + +```bash +pip install semantic-kernel +``` + +> [!IMPORTANT] +> Depending upon which AI Service you use as part of the `ChatCompletionAgent`, you may need to install extra packages. Please check for the required extra on the following [page](../../concepts/ai-services/chat-completion/index.md#creating-a-chat-completion-service) + ::: zone-end @@ -73,12 +73,12 @@ Onnx|[`Microsoft.SemanticKernel.Connectors.Onnx`](/dotnet/api/microsoft.semantic ::: zone-end +## Creating a `ChatCompletionAgent` -## Creating a Chat Completion Agent - -A _chat completion agent_ is fundamentally based on an [AI services](../../concepts/ai-services/index.md). As such, creating an _chat completion agent_ starts with creating a [`Kernel`](../../concepts/kernel.md) instance that contains one or more chat-completion services and then instantiating the agent with a reference to that [`Kernel`](../../concepts/kernel.md) instance. +A `ChatCompletionAgent` is fundamentally based on an [AI services](../../concepts/ai-services/index.md). As such, creating a `ChatCompletionAgent` starts with creating a [`Kernel`](../../concepts/kernel.md) instance that contains one or more chat-completion services and then instantiating the agent with a reference to that [`Kernel`](../../concepts/kernel.md) instance. ::: zone pivot="programming-language-csharp" + ```csharp // Initialize a Kernel with a chat-completion service IKernelBuilder builder = Kernel.CreateBuilder(); @@ -96,23 +96,43 @@ ChatCompletionAgent agent = Kernel = kernel }; ``` + ::: zone-end ::: zone pivot="programming-language-python" +There are two ways to create a `ChatCompletionAgent`: + +### 1. By providing the chat completion service directly + ```python -# Define the Kernel +from semantic_kernel.agents import ChatCompletionAgent + +# Create the agent by directly providing the chat completion service +agent = ChatCompletionAgent( + service=AzureChatCompletion(), # your chat completion service instance + name="", + instructions="", +) +``` + +### 2. By creating a Kernel first, adding the service to it, then providing the kernel + +```python +# Define the kernel kernel = Kernel() -# Add the AzureChatCompletion AI Service to the Kernel +# Add the chat completion service to the kernel kernel.add_service(AzureChatCompletion()) -# Create the agent +# Create the agent using the kernel agent = ChatCompletionAgent( kernel=kernel, name="", instructions="", ) ``` + +The first method is useful when you already have a chat completion service ready. The second method is beneficial when you need a kernel that manages multiple services or additional functionalities. ::: zone-end ::: zone pivot="programming-language-java" @@ -121,14 +141,14 @@ agent = ChatCompletionAgent( ::: zone-end - ## AI Service Selection -No different from using _Semantic Kernel_ [AI services](../../concepts/ai-services/index.md) directly, a `ChatCompletionAgent` supports the specification of a _service-selector_. A _service-selector_ identifies which [AI service](../../concepts/ai-services/index.md) to target when the [`Kernel`](../../concepts/kernel.md) contains more than one. +No different from using Semantic Kernel [AI services](../../concepts/ai-services/index.md) directly, a `ChatCompletionAgent` supports the specification of a service-selector. A service-selector identifies which [AI service](../../concepts/ai-services/index.md) to target when the [`Kernel`](../../concepts/kernel.md) contains more than one. -> Note: If multiple [AI services](../../concepts/ai-services/index.md) are present and no _service-selector_ is provided, the same _default_ logic is applied for the agent that you'd find when using an [AI services](../../concepts/ai-services/index.md) outside of the `Agent Framework` +> Note: If multiple [AI services](../../concepts/ai-services/index.md) are present and no service-selector is provided, the same default logic is applied for the agent that you'd find when using an [AI services](../../concepts/ai-services/index.md) outside of the `Agent Framework` ::: zone pivot="programming-language-csharp" + ```csharp IKernelBuilder builder = Kernel.CreateBuilder(); @@ -149,12 +169,14 @@ ChatCompletionAgent agent = new OpenAIPromptExecutionSettings() { ServiceId = "service-2" // The target service-identifier. - }); + }) }; ``` + ::: zone-end ::: zone pivot="programming-language-python" + ```python from semantic_kernel.connectors.ai.open_ai import ( AzureChatCompletion, @@ -178,6 +200,7 @@ agent = ChatCompletionAgent( arguments=KernelArguments(settings=settings) ) ``` + ::: zone-end ::: zone pivot="programming-language-java" @@ -190,24 +213,40 @@ agent = ChatCompletionAgent( ::: zone pivot="programming-language-csharp" -Conversing with your `ChatCompletionAgent` is based on a `ChatHistory` instance, no different from interacting with a _Chat Completion_ [AI service](../../concepts/ai-services/index.md). +Conversing with your `ChatCompletionAgent` is based on a `ChatHistory` instance, no different from interacting with a Chat Completion [AI service](../../concepts/ai-services/index.md). + +You can simply invoke the agent with your user message. ```csharp // Define agent ChatCompletionAgent agent = ...; -// Create a ChatHistory object to maintain the conversation state. -ChatHistory chat = []; +// Generate the agent response(s) +await foreach (ChatMessageContent response in agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, ""))) +{ + // Process agent response(s)... +} +``` + +You can also use an `AgentThread` to have a conversation with your agent. +Here we are using a `ChatHistoryAgentThread`. + +The `ChatHistoryAgentThread` can also take an optional `ChatHistory` +object as input, via its constructor, if resuming a previous conversation. (not shown) + +```csharp +// Define agent +ChatCompletionAgent agent = ...; -// Add a user message to the conversation -chat.Add(new ChatMessageContent(AuthorRole.User, "")); +AgentThread thread = new ChatHistoryAgentThread(); // Generate the agent response(s) -await foreach (ChatMessageContent response in agent.InvokeAsync(chat)) +await foreach (ChatMessageContent response in agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, ""), thread)) { // Process agent response(s)... } ``` + ::: zone-end ::: zone pivot="programming-language-python" @@ -219,31 +258,39 @@ The easiest is to call and await `get_response`: ```python # Define agent agent = ChatCompletionAgent(...) - -# Define the chat history -chat = ChatHistory() - -# Add the user message -chat.add_message(ChatMessageContent(role=AuthorRole.USER, content=input)) # Generate the agent response -response = await agent.get_response(chat) -# response is a `ChatMessageContent` object +response = await agent.get_response(messages="user input") +# response is an `AgentResponseItem[ChatMessageContent]` object ``` -Otherwise, calling the `invoke` method returns an `AsyncIterable` of `ChatMessageContent`. + +If you want the agent to maintain conversation history between invocations, you can pass it a `ChatHistoryAgentThread` as follows: ```python + # Define agent agent = ChatCompletionAgent(...) - -# Define the chat history -chat = ChatHistory() -# Add the user message -chat.add_user_message(ChatMessageContent(role=AuthorRole.USER, content=input)) +# Generate the agent response(s) +response = await agent.get_response(messages="user input") + +# Generate another response, continuing the conversation thread from the first response. +response2 = await agent.get_response(messages="user input", thread=response.thread) +# process agent response(s) + +``` + +Calling the `invoke` method returns an `AsyncIterable` of `AgentResponseItem[ChatMessageContent]`. + +```python +# Define agent +agent = ChatCompletionAgent(...) + +# Define the thread +thread = ChatHistoryAgentThread() # Generate the agent response(s) -async for response in agent.invoke(chat): +async for response in agent.invoke(messages="user input", thread=thread): # process agent response(s) ``` @@ -253,14 +300,11 @@ The `ChatCompletionAgent` also supports streaming in which the `invoke_stream` m # Define agent agent = ChatCompletionAgent(...) -# Define the chat history -chat = ChatHistory() - -# Add the user message -chat.add_message(ChatMessageContent(role=AuthorRole.USER, content=input)) +# Define the thread +thread = ChatHistoryAgentThread() # Generate the agent response(s) -async for response in agent.invoke_stream(chat): +async for response in agent.invoke_stream(messages="user input", thread=thread): # process agent response(s) ``` @@ -272,13 +316,151 @@ async for response in agent.invoke_stream(chat): ::: zone-end +## Handling Intermediate Messages with a `ChatCompletionAgent` -#### How-To: +The Semantic Kernel `ChatCompletionAgent` is designed to invoke an agent that fulfills user queries or questions. During invocation, the agent may execute tools to derive the final answer. To access intermediate messages produced during this process, callers can supply a callback function that handles instances of `FunctionCallContent` or `FunctionResultContent`. + +::: zone pivot="programming-language-csharp" +> Callback documentation for the `ChatCompletionAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Configuring the `on_intermediate_message` callback within `agent.invoke(...)` or `agent.invoke_stream(...)` allows the caller to receive intermediate messages generated during the process of formulating the agent's final response. + +```python +import asyncio +from typing import Annotated + +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion +from semantic_kernel.contents import AuthorRole, ChatMessageContent, FunctionCallContent, FunctionResultContent +from semantic_kernel.functions import kernel_function + +class MenuPlugin: + """A sample Menu Plugin used for the concept sample.""" + + @kernel_function(description="Provides a list of specials from the menu.") + def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]: + return """ + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + """ + + @kernel_function(description="Provides the price of the requested menu item.") + def get_item_price( + self, menu_item: Annotated[str, "The name of the menu item."] + ) -> Annotated[str, "Returns the price of the menu item."]: + return "$9.99" + +# Define a list to hold callback message content +intermediate_steps: list[ChatMessageContent] = [] + +# Define an async method to handle the `on_intermediate_message` callback +async def handle_intermediate_steps(message: ChatMessageContent) -> None: + intermediate_steps.append(message) + +async def main(): + # Create the ChatCompletionAgent instance + agent = ChatCompletionAgent( + service=AzureChatCompletion(), + instructions="your instructions", + name="name", + plugins=[MenuPlugin()], + ) + + user_inputs = [ + "Hello", + "What is the special soup?", + "What is the special drink?", + "How much is that?", + "Thank you", + ] + + thread = None + + # Generate the agent response(s) + for user_input in user_inputs: + print(f"# {AuthorRole.USER}: '{user_input}'") + async for response in agent.invoke( + messages=user_input, + thread=thread, + on_intermediate_message=handle_intermediate_steps, + ): + thread = response.thread + print(f"# {response.name}: {response.content}") + + # Delete the thread when it is no longer needed + await thread.delete() if thread else None + + # Print the intermediate steps + print("\nIntermediate Steps:") + for msg in intermediate_steps: + if any(isinstance(item, FunctionResultContent) for item in msg.items): + for fr in msg.items: + if isinstance(fr, FunctionResultContent): + print(f"Function Result:> {fr.result} for function: {fr.name}") + elif any(isinstance(item, FunctionCallContent) for item in msg.items): + for fcc in msg.items: + if isinstance(fcc, FunctionCallContent): + print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}") + else: + print(f"{msg.role}: {msg.content}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + +The following demonstrates sample output from the agent invocation process: + +```bash +Sample Output: + +# AuthorRole.USER: 'Hello' +# Host: Hi there! How can I assist you with the menu today? +# AuthorRole.USER: 'What is the special soup?' +# Host: The special soup is Clam Chowder. +# AuthorRole.USER: 'What is the special drink?' +# Host: The special drink is Chai Tea. +# AuthorRole.USER: 'How much is that?' +# Host: Could you please specify the menu item you are asking about? +# AuthorRole.USER: 'Thank you' +# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. + +Intermediate Steps: +AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today? +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special soup is Clam Chowder. +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special drink is Chai Tea. +AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about? +AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### How-To For an end-to-end example for a `ChatCompletionAgent`, see: - [How-To: `ChatCompletionAgent`](./examples/example-chat-agent.md) - > [!div class="nextstepaction"] -> [Exploring `OpenAIAssistantAgent`](./assistant-agent.md) +> [Explore the OpenAI Assistant Agent](./assistant-agent.md) diff --git a/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md b/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md index 4329e493..6a7dd1a5 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md +++ b/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md @@ -11,14 +11,14 @@ ms.service: semantic-kernel # How-To: Coordinate Agent Collaboration using Agent Group Chat > [!IMPORTANT] -> This feature is in the experimental stage. Features at this stage are still under development and subject to change before advancing to the preview or release candidate stage. +> This feature is in the experimental stage. Features at this stage are under development and subject to change before advancing to the preview or release candidate stage. ## Overview In this sample, we will explore how to use `AgentGroupChat` to coordinate collaboration of two different agents working to review and rewrite user provided content. Each agent is assigned a distinct role: -- **Reviewer**: Reviews and provides direction to _Writer_. -- **Writer**: Updates user content based on _Reviewer_ input. +- **Reviewer**: Reviews and provides direction to Writer. +- **Writer**: Updates user content based on Reviewer input. The approach will be broken down step-by-step to high-light the key parts of the coding process. @@ -31,7 +31,7 @@ Before proceeding with feature coding, make sure your development environment is > [!TIP] > This sample uses an optional text file as part of processing. If you'd like to use it, you may download it [here](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/LearnResources/Resources/WomensSuffrage.txt). Place the file in your code working directory. -Start by creating a _Console_ project. Then, include the following package references to ensure all required dependencies are available. +Start by creating a Console project. Then, include the following package references to ensure all required dependencies are available. To add package dependencies from the command-line use the `dotnet` command: @@ -45,7 +45,7 @@ dotnet add package Microsoft.SemanticKernel.Connectors.AzureOpenAI dotnet add package Microsoft.SemanticKernel.Agents.Core --prerelease ``` -> If managing _NuGet_ packages in _Visual Studio_, ensure `Include prerelease` is checked. +> If managing NuGet packages in Visual Studio, ensure `Include prerelease` is checked. The project file (`.csproj`) should contain the following `PackageReference` definitions: @@ -109,7 +109,7 @@ from semantic_kernel.functions import KernelFunctionFromPrompt ::: zone pivot="programming-language-csharp" -This sample requires configuration setting in order to connect to remote services. You will need to define settings for either _OpenAI_ or _Azure OpenAI_. +This sample requires configuration setting in order to connect to remote services. You will need to define settings for either OpenAI or Azure OpenAI. ```powershell # OpenAI @@ -391,7 +391,7 @@ RULES: ::: zone-end ::: zone pivot="programming-language-csharp" -The _Writer_ agent is similiar, but doesn't require the specification of _Execution Settings_ since it isn't configured with a plug-in. +The _Writer_ agent is similiar, but doesn't require the specification of Execution Settings since it isn't configured with a plug-in. Here the _Writer_ is given a single-purpose task, follow direction and rewrite the content. @@ -862,8 +862,6 @@ Try using these suggested inputs: 8. its good, but is it ready for my college professor? ```csharp -// Copyright (c) Microsoft. All rights reserved. - using System; using System.ComponentModel; using System.Diagnostics; @@ -1131,8 +1129,6 @@ You can try using one of the suggested inputs. As the agent chat begins, the age > You can reference any file by providing `@`. To reference the "WomensSuffrage" text from above, download it [here](https://github.com/microsoft/semantic-kernel/blob/main/python/samples/learn_resources/resources/WomensSuffrage.txt) and place it in your current working directory. You can then reference it with `@WomensSuffrage.txt`. ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import os diff --git a/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md b/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md index be6fdc4d..1f51670c 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md +++ b/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md @@ -15,7 +15,7 @@ ms.service: semantic-kernel ## Overview -In this sample, we will explore how to use the _code-interpreter_ tool of an [`OpenAIAssistantAgent`](../assistant-agent.md) to complete data-analysis tasks. The approach will be broken down step-by-step to high-light the key parts of the coding process. As part of the task, the agent will generate both image and text responses. This will demonstrate the versatility of this tool in performing quantitative analysis. +In this sample, we will explore how to use the code-interpreter tool of an [`OpenAIAssistantAgent`](../assistant-agent.md) to complete data-analysis tasks. The approach will be broken down step-by-step to high-light the key parts of the coding process. As part of the task, the agent will generate both image and text responses. This will demonstrate the versatility of this tool in performing quantitative analysis. Streaming will be used to deliver the agent's responses. This will provide real-time updates as the task progresses. @@ -26,7 +26,7 @@ Before proceeding with feature coding, make sure your development environment is ::: zone pivot="programming-language-csharp" -Start by creating a _Console_ project. Then, include the following package references to ensure all required dependencies are available. +Start by creating a Console project. Then, include the following package references to ensure all required dependencies are available. To add package dependencies from the command-line use the `dotnet` command: @@ -40,7 +40,7 @@ dotnet add package Microsoft.SemanticKernel dotnet add package Microsoft.SemanticKernel.Agents.OpenAI --prerelease ``` -> If managing _NuGet_ packages in _Visual Studio_, ensure `Include prerelease` is checked. +> If managing NuGet packages in Visual Studio, ensure `Include prerelease` is checked. The project file (`.csproj`) should contain the following `PackageReference` definitions: @@ -64,7 +64,7 @@ The `Agent Framework` is experimental and requires warning suppression. This ma ``` -Additionally, copy the `PopulationByAdmin1.csv` and `PopulationByCountry.csv` data files from [_Semantic Kernel_ `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Resources). Add these files in your project folder and configure to have them copied to the output directory: +Additionally, copy the `PopulationByAdmin1.csv` and `PopulationByCountry.csv` data files from [Semantic Kernel `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Resources). Add these files in your project folder and configure to have them copied to the output directory: ```xml @@ -86,11 +86,11 @@ Start by creating a folder that will hold your script (`.py` file) and the sampl import asyncio import os -from semantic_kernel.agents.open_ai import AzureAssistantAgent +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent from semantic_kernel.contents import StreamingFileReferenceContent ``` -Additionally, copy the `PopulationByAdmin1.csv` and `PopulationByCountry.csv` data files from the [_Semantic Kernel_ `learn_resources/resources` directory](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/resources). Add these files to your working directory. +Additionally, copy the `PopulationByAdmin1.csv` and `PopulationByCountry.csv` data files from the [Semantic Kernel `learn_resources/resources` directory](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/resources). Add these files to your working directory. ::: zone-end @@ -102,7 +102,7 @@ Additionally, copy the `PopulationByAdmin1.csv` and `PopulationByCountry.csv` da ## Configuration -This sample requires configuration setting in order to connect to remote services. You will need to define settings for either _OpenAI_ or _Azure OpenAI_. +This sample requires configuration setting in order to connect to remote services. You will need to define settings for either OpenAI or Azure OpenAI. ::: zone pivot="programming-language-csharp" @@ -198,7 +198,7 @@ The coding process for this sample involves: 1. [Setup](#setup) - Initializing settings and the plug-in. 2. [Agent Definition](#agent-definition) - Create the _OpenAI_Assistant`Agent` with templatized instructions and plug-in. -3. [The _Chat_ Loop](#the-chat-loop) - Write the loop that drives user / agent interaction. +3. [The Chat Loop](#the-chat-loop) - Write the loop that drives user / agent interaction. The full example code is provided in the [Final](#final) section. Refer to that section for the complete implementation. @@ -208,13 +208,12 @@ The full example code is provided in the [Final](#final) section. Refer to that Prior to creating an `OpenAIAssistantAgent`, ensure the configuration settings are available and prepare the file resources. -Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `OpenAIClientProvider` that will be used for the [Agent Definition](#agent-definition) as well as file-upload. +Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `AzureOpenAIClient` that will be used for the [Agent Definition](#agent-definition) as well as file-upload. ```csharp Settings settings = new(); -OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); ``` ::: zone-end @@ -226,11 +225,11 @@ OpenAIClientProvider clientProvider = ::: zone pivot="programming-language-csharp" -Use the `OpenAIClientProvider` to access an `OpenAIFileClient` and upload the two data-files described in the previous [Configuration](#configuration) section, preserving the _File Reference_ for final clean-up. +Use the `AzureOpenAIClient` to access an `OpenAIFileClient` and upload the two data-files described in the previous [Configuration](#configuration) section, preserving the _File Reference_ for final clean-up. ```csharp Console.WriteLine("Uploading files..."); -OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); +OpenAIFileClient fileClient = client.GetOpenAIFileClient(); OpenAIFile fileDataCountryDetail = await fileClient.UploadFileAsync("PopulationByAdmin1.csv", FileUploadPurpose.Assistants); OpenAIFile fileDataCountryList = await fileClient.UploadFileAsync("PopulationByCountry.csv", FileUploadPurpose.Assistants); ``` @@ -303,27 +302,27 @@ We first set up the Azure OpenAI resources to obtain the client and model. Next, ::: zone pivot="programming-language-csharp" -We are now ready to instantiate an `OpenAIAssistantAgent`. The agent is configured with its target model, _Instructions_, and the _Code Interpreter_ tool enabled. Additionally, we explicitly associate the two data files with the _Code Interpreter_ tool. +We are now ready to instantiate an `OpenAIAssistantAgent` by first creating an assistant definition. The assistant is configured with its target model, _Instructions_, and the _Code Interpreter_ tool enabled. Additionally, we explicitly associate the two data files with the _Code Interpreter_ tool. ```csharp Console.WriteLine("Defining agent..."); -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = - """ - Analyze the available data to provide an answer to the user's question. - Always format response using markdown. - Always include a numerical index that starts at 1 for any lists or tables. - Always sort lists in ascending order. - """, - EnableCodeInterpreter = true, - CodeInterpreterFileIds = [fileDataCountryList.Id, fileDataCountryDetail.Id], - }, - new Kernel()); +AssistantClient assistantClient = client.GetAssistantClient(); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: + """ + Analyze the available data to provide an answer to the user's question. + Always format response using markdown. + Always include a numerical index that starts at 1 for any lists or tables. + Always sort lists in ascending order. + """, + enableCodeInterpreter: true, + codeInterpreterFileIds: [fileDataCountryList.Id, fileDataCountryDetail.Id]); + +// Create agent +OpenAIAssistantAgent agent = new(assistant, assistantClient); ``` ::: zone-end @@ -346,16 +345,16 @@ agent = AzureAssistantAgent( ::: zone-end -### The _Chat_ Loop +### The Chat Loop -At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating an _Assistant Thread_ to maintain the conversation state and creating an empty loop. +At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating an `AgentThread` to maintain the conversation state and creating an empty loop. Let's also ensure the resources are removed at the end of execution to minimize unnecessary charges. ::: zone pivot="programming-language-csharp" ```csharp Console.WriteLine("Creating thread..."); -string threadId = await agent.CreateThreadAsync(); +AssistantAgentThread agentThread = new(); Console.WriteLine("Ready!"); @@ -374,8 +373,8 @@ finally Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + agentThread.DeleteAsync(), + assistantClient.DeleteAssistantAsync(assistant.Id), fileClient.DeleteFileAsync(fileDataCountryList.Id), fileClient.DeleteFileAsync(fileDataCountryDetail.Id), ]); @@ -385,8 +384,7 @@ finally ::: zone pivot="programming-language-python" ```python -print("Creating thread...") -thread_id = await agent.create_thread() +thread: AssistantAgentThread = None try: is_complete: bool = False @@ -396,7 +394,7 @@ try: finally: print("\nCleaning up resources...") [await client.files.delete(file_id) for file_id in file_ids] - await client.beta.threads.delete(thread.id) + await thread.delete() if thread else None await client.beta.assistants.delete(agent.id) ``` ::: zone-end @@ -407,7 +405,7 @@ finally: ::: zone-end -Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. Valid input will be added to the _Assistant Thread_ as a _User_ message. +Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. ::: zone pivot="programming-language-csharp" ```csharp @@ -424,7 +422,7 @@ if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase)) break; } -await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); +var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); ``` @@ -439,8 +437,6 @@ if not user_input: if user_input.lower() == "exit": is_complete = True break - -await agent.add_chat_message(thread_id=thread_id, message=ChatMessageContent(role=AuthorRole.USER, content=user_input)) ``` ::: zone-end @@ -537,12 +533,12 @@ async def download_response_image(agent, file_ids: list[str]): ::: zone-end -To generate an `Agent` response to user input, invoke the agent by specifying the _Assistant Thread_. In this example, we choose a streamed response and capture any generated _File References_ for download and review at the end of the response cycle. It's important to note that generated code is identified by the presence of a _Metadata_ key in the response message, distinguishing it from the conversational reply. +To generate an `Agent` response to user input, invoke the agent by providing the message and the `AgentThread`. In this example, we choose a streamed response and capture any generated _File References_ for download and review at the end of the response cycle. It's important to note that generated code is identified by the presence of a _Metadata_ key in the response message, distinguishing it from the conversational reply. ::: zone pivot="programming-language-csharp" ```csharp bool isCode = false; -await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(threadId)) +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) { if (isCode != (response.Metadata?.ContainsKey(OpenAIAssistantAgent.CodeInterpreterMetadataKey) ?? false)) { @@ -566,18 +562,31 @@ fileIds.Clear(); ::: zone pivot="programming-language-python" ```python -is_code: bool = False -async for response in agent.invoke(stream(thread_id=thread_id): - if is_code != metadata.get("code"): - print() - is_code = not is_code - - print(f"{response.content}) - - file_ids.extend( - [item.file_id for item in response.items if isinstance(item, StreamingFileReferenceContent)] - ) - +is_code = False +last_role = None +async for response in agent.invoke_stream(messages=user_input, thread=thread): + current_is_code = response.metadata.get("code", False) + + if current_is_code: + if not is_code: + print("\n\n```python") + is_code = True + print(response.content, end="", flush=True) + else: + if is_code: + print("\n```") + is_code = False + last_role = None + if hasattr(response, "role") and response.role is not None and last_role != response.role: + print(f"\n# {response.role}: ", end="", flush=True) + last_role = response.role + print(response.content, end="", flush=True) + file_ids.extend([ + item.file_id for item in response.items if isinstance(item, StreamingFileReferenceContent) + ]) + thread = response.thread +if is_code: + print("```\n") print() await download_response_image(agent, file_ids) @@ -604,17 +613,19 @@ Try using these suggested inputs: ::: zone pivot="programming-language-csharp" ```csharp +using Azure.AI.OpenAI; +using Azure.Identity; +using Microsoft.SemanticKernel; +using Microsoft.SemanticKernel.Agents.OpenAI; +using Microsoft.SemanticKernel.ChatCompletion; +using OpenAI.Assistants; +using OpenAI.Files; using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; using System.Threading.Tasks; -using Azure.Identity; -using Microsoft.SemanticKernel; -using Microsoft.SemanticKernel.Agents.OpenAI; -using Microsoft.SemanticKernel.ChatCompletion; -using OpenAI.Files; namespace AgentsSample; @@ -625,35 +636,39 @@ public static class Program // Load configuration from environment variables or user secrets. Settings settings = new(); - OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + // Initialize the clients + AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + //OpenAIClient client = OpenAIAssistantAgent.CreateOpenAIClient(new ApiKeyCredential(settings.OpenAI.ApiKey))); + AssistantClient assistantClient = client.GetAssistantClient(); + OpenAIFileClient fileClient = client.GetOpenAIFileClient(); + // Upload files Console.WriteLine("Uploading files..."); - OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); OpenAIFile fileDataCountryDetail = await fileClient.UploadFileAsync("PopulationByAdmin1.csv", FileUploadPurpose.Assistants); OpenAIFile fileDataCountryList = await fileClient.UploadFileAsync("PopulationByCountry.csv", FileUploadPurpose.Assistants); - Console.WriteLine("Defining agent..."); - OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = + // Define assistant + Console.WriteLine("Defining assistant..."); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ Analyze the available data to provide an answer to the user's question. Always format response using markdown. Always include a numerical index that starts at 1 for any lists or tables. Always sort lists in ascending order. """, - EnableCodeInterpreter = true, - CodeInterpreterFileIds = [fileDataCountryList.Id, fileDataCountryDetail.Id], - }, - new Kernel()); + enableCodeInterpreter: true, + codeInterpreterFileIds: [fileDataCountryList.Id, fileDataCountryDetail.Id]); + + // Create agent + OpenAIAssistantAgent agent = new(assistant, assistantClient); + // Create the conversation thread Console.WriteLine("Creating thread..."); - string threadId = await agent.CreateThreadAsync(); + AssistantAgentThread agentThread = new(); Console.WriteLine("Ready!"); @@ -676,12 +691,12 @@ public static class Program break; } - await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); + var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); bool isCode = false; - await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(threadId)) + await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(message, agentThread)) { if (isCode != (response.Metadata?.ContainsKey(OpenAIAssistantAgent.CodeInterpreterMetadataKey) ?? false)) { @@ -709,8 +724,8 @@ public static class Program Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + agentThread.DeleteAsync(), + assistantClient.DeleteAssistantAsync(assistant.Id), fileClient.DeleteFileAsync(fileDataCountryList.Id), fileClient.DeleteFileAsync(fileDataCountryDetail.Id), ]); @@ -761,13 +776,11 @@ public static class Program ::: zone pivot="programming-language-python" ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import logging import os -from semantic_kernel.agents.open_ai import AzureAssistantAgent +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent from semantic_kernel.contents import StreamingFileReferenceContent logging.basicConfig(level=logging.ERROR) @@ -776,7 +789,7 @@ logging.basicConfig(level=logging.ERROR) The following sample demonstrates how to create a simple, OpenAI assistant agent that utilizes the code interpreter to analyze uploaded files. -""" +""" # Let's form the file paths that we will later pass to the assistant csv_file_path_1 = os.path.join( @@ -858,8 +871,7 @@ async def main(): definition=definition, ) - print("Creating thread...") - thread = await client.beta.threads.create() + thread: AssistantAgentThread = None try: is_complete: bool = False @@ -873,11 +885,9 @@ async def main(): is_complete = True break - await agent.add_chat_message(thread_id=thread.id, message=user_input) - is_code = False last_role = None - async for response in agent.invoke_stream(thread_id=thread.id): + async for response in agent.invoke_stream(messages=user_input, thread=thread): current_is_code = response.metadata.get("code", False) if current_is_code: @@ -897,6 +907,7 @@ async def main(): file_ids.extend([ item.file_id for item in response.items if isinstance(item, StreamingFileReferenceContent) ]) + thread = response.thread if is_code: print("```\n") print() @@ -907,7 +918,7 @@ async def main(): finally: print("\nCleaning up resources...") [await client.files.delete(file_id) for file_id in file_ids] - await client.beta.threads.delete(thread.id) + await thread.delete() if thread else None await client.beta.assistants.delete(agent.id) diff --git a/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md b/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md index 001395a6..c40b77a6 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md +++ b/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md @@ -15,7 +15,7 @@ ms.service: semantic-kernel ## Overview -In this sample, we will explore how to use the _file-search_ tool of an [`OpenAIAssistantAgent`](../assistant-agent.md) to complete comprehension tasks. The approach will be step-by-step, ensuring clarity and precision throughout the process. As part of the task, the agent will provide document citations within the response. +In this sample, we will explore how to use the file-search tool of an [`OpenAIAssistantAgent`](../assistant-agent.md) to complete comprehension tasks. The approach will be step-by-step, ensuring clarity and precision throughout the process. As part of the task, the agent will provide document citations within the response. Streaming will be used to deliver the agent's responses. This will provide real-time updates as the task progresses. @@ -38,7 +38,7 @@ dotnet add package Microsoft.SemanticKernel dotnet add package Microsoft.SemanticKernel.Agents.OpenAI --prerelease ``` -> If managing _NuGet_ packages in _Visual Studio_, ensure `Include prerelease` is checked. +> If managing NuGet packages in Visual Studio, ensure `Include prerelease` is checked. The project file (`.csproj`) should contain the following `PackageReference` definitions: @@ -62,7 +62,7 @@ The `Agent Framework` is experimental and requires warning suppression. This ma ``` -Additionally, copy the `Grimms-The-King-of-the-Golden-Mountain.txt`, `Grimms-The-Water-of-Life.txt` and `Grimms-The-White-Snake.txt` public domain content from [_Semantic Kernel_ `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Resources). Add these files in your project folder and configure to have them copied to the output directory: +Additionally, copy the `Grimms-The-King-of-the-Golden-Mountain.txt`, `Grimms-The-Water-of-Life.txt` and `Grimms-The-White-Snake.txt` public domain content from [Semantic Kernel `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Resources). Add these files in your project folder and configure to have them copied to the output directory: ```xml @@ -86,11 +86,11 @@ Start by creating a folder that will hold your script (`.py` file) and the sampl import asyncio import os -from semantic_kernel.agents.open_ai import AzureAssistantAgent +from semantic_kernel.agents import AssistantAgentThread, AzureAssistantAgent from semantic_kernel.contents import StreamingAnnotationContent ``` -Additionally, copy the `Grimms-The-King-of-the-Golden-Mountain.txt`, `Grimms-The-Water-of-Life.txt` and `Grimms-The-White-Snake.txt` public domain content from [_Semantic Kernel_ `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/resources). Add these files in your project folder. +Additionally, copy the `Grimms-The-King-of-the-Golden-Mountain.txt`, `Grimms-The-Water-of-Life.txt` and `Grimms-The-White-Snake.txt` public domain content from [Semantic Kernel `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/resources). Add these files in your project folder. ::: zone-end @@ -103,7 +103,7 @@ Additionally, copy the `Grimms-The-King-of-the-Golden-Mountain.txt`, `Grimms-The ## Configuration -This sample requires configuration setting in order to connect to remote services. You will need to define settings for either _OpenAI_ or _Azure OpenAI_. +This sample requires configuration setting in order to connect to remote services. You will need to define settings for either OpenAI or Azure OpenAI. ::: zone pivot="programming-language-csharp" @@ -209,16 +209,12 @@ Prior to creating an `OpenAIAssistantAgent`, ensure the configuration settings a ::: zone pivot="programming-language-csharp" -Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `OpenAIClientProvider` that will be used for the [Agent Definition](#agent-definition) as well as file-upload and the creation of a `VectorStore`. +Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `AzureOpenAIClient` that will be used for the [Agent Definition](#agent-definition) as well as file-upload and the creation of a `VectorStore`. ```csharp - Settings settings = new(); -OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI( - new AzureCliCredential(), - new Uri(settings.AzureOpenAI.Endpoint)); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); ``` ::: zone-end @@ -242,11 +238,11 @@ Now create an empty _Vector Store for use with the _File Search_ tool: ::: zone pivot="programming-language-csharp" -Use the `OpenAIClientProvider` to access a `VectorStoreClient` and create a `VectorStore`. +Use the `AzureOpenAIClient` to access a `VectorStoreClient` and create a `VectorStore`. ```csharp Console.WriteLine("Creating store..."); -VectorStoreClient storeClient = clientProvider.Client.GetVectorStoreClient(); +VectorStoreClient storeClient = client.GetVectorStoreClient(); CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true); string storeId = operation.VectorStoreId; ``` @@ -317,7 +313,7 @@ Now upload those files and add them to the _Vector Store_ by using the previousl Dictionary fileReferences = []; Console.WriteLine("Uploading files..."); -OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); +OpenAIFileClient fileClient = client.GetOpenAIFileClient(); foreach (string fileName in _fileNames) { OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants); @@ -339,27 +335,26 @@ We are now ready to instantiate an `OpenAIAssistantAgent`. The agent is configur ::: zone pivot="programming-language-csharp" -We will utilize the `OpenAIClientProvider` again as part of creating the `OpenAIAssistantAgent`: +We will utilize the `AzureOpenAIClient` again as part of creating the `OpenAIAssistantAgent`: ```csharp -Console.WriteLine("Defining agent..."); -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = +Console.WriteLine("Defining assistant..."); +Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ The document store contains the text of fictional stories. Always analyze the document store to provide an answer to the user's question. Never rely on your knowledge of stories not included in the document store. Always format response using markdown. """, - EnableFileSearch = true, - VectorStoreId = storeId, - }, - new Kernel()); + enableFileSearch: true, + vectorStoreId: storeId); + +// Create agent +OpenAIAssistantAgent agent = new(assistant, assistantClient); ``` ::: zone-end @@ -395,14 +390,14 @@ agent = AzureAssistantAgent( ### The _Chat_ Loop -At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating an _Assistant Thread_ to maintain the conversation state and creating an empty loop. +At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating an `AgentThread` to maintain the conversation state and creating an empty loop. Let's also ensure the resources are removed at the end of execution to minimize unnecessary charges. ::: zone pivot="programming-language-csharp" ```csharp Console.WriteLine("Creating thread..."); -string threadId = await agent.CreateThreadAsync(); +OpenAIAssistantAgent agentThread = new(); Console.WriteLine("Ready!"); @@ -420,8 +415,8 @@ finally Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + agentThread.DeleteAsync(); + assistantClient.DeleteAssistantAsync(assistant.Id), storeClient.DeleteVectorStoreAsync(storeId), ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key)) ]); @@ -454,7 +449,7 @@ finally: ::: zone-end -Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. Valid input will be added to the _Assistant Thread_ as a _User_ message. +Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. ::: zone pivot="programming-language-csharp" ```csharp @@ -471,7 +466,7 @@ if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase)) break; } -await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); +var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); ``` ::: zone-end @@ -485,10 +480,6 @@ if not user_input: if user_input.lower() == "exit": is_complete = True break - -await agent.add_chat_message( - thread_id=thread_id, message=ChatMessageContent(role=AuthorRole.USER, content=user_input) -) ``` ::: zone-end @@ -519,12 +510,12 @@ private static string ReplaceUnicodeBrackets(this string content) => ::: zone-end -To generate an `Agent` response to user input, invoke the agent by specifying the _Assistant Thread_. In this example, we choose a streamed response and capture any associated _Citation Annotations_ for display at the end of the response cycle. Note each streamed chunk is being reformatted using the previous helper method. +To generate an `Agent` response to user input, invoke the agent by specifying the message and agent thread. In this example, we choose a streamed response and capture any associated _Citation Annotations_ for display at the end of the response cycle. Note each streamed chunk is being reformatted using the previous helper method. ::: zone pivot="programming-language-csharp" ```csharp List footnotes = []; -await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(threadId)) +await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(message, agentThread)) { // Capture annotations for footnotes footnotes.AddRange(chunk.Items.OfType()); @@ -550,7 +541,8 @@ if (footnotes.Count > 0) ::: zone pivot="programming-language-python" ```python footnotes: list[StreamingAnnotationContent] = [] -async for response in agent.invoke_stream(thread_id=thread_id): +async for response in agent.invoke_stream(messages=user_input, thread=thread): + thread = response.thread footnotes.extend([item for item in response.items if isinstance(item, StreamingAnnotationContent)]) print(f"{response.content}", end="", flush=True) @@ -585,16 +577,19 @@ Try using these suggested inputs: ::: zone pivot="programming-language-csharp" ```csharp -using System; -using System.Collections.Generic; -using System.Linq; -using System.Threading.Tasks; +using Azure.AI.OpenAI; using Azure.Identity; using Microsoft.SemanticKernel; +using Microsoft.SemanticKernel.Agents; using Microsoft.SemanticKernel.Agents.OpenAI; using Microsoft.SemanticKernel.ChatCompletion; +using OpenAI.Assistants; using OpenAI.Files; using OpenAI.VectorStores; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; namespace AgentsSample; @@ -616,21 +611,21 @@ public static class Program // Load configuration from environment variables or user secrets. Settings settings = new(); - OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI( - new AzureCliCredential(), - new Uri(settings.AzureOpenAI.Endpoint)); + // Initialize the clients + AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + //OpenAIClient client = OpenAIAssistantAgent.CreateOpenAIClient(new ApiKeyCredential(settings.OpenAI.ApiKey))); + AssistantClient assistantClient = client.GetAssistantClient(); + OpenAIFileClient fileClient = client.GetOpenAIFileClient(); + VectorStoreClient storeClient = client.GetVectorStoreClient(); + // Create the vector store Console.WriteLine("Creating store..."); - VectorStoreClient storeClient = clientProvider.Client.GetVectorStoreClient(); CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true); string storeId = operation.VectorStoreId; - // Retain file references. - Dictionary fileReferences = []; - + // Upload files and retain file references. Console.WriteLine("Uploading files..."); - OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); + Dictionary fileReferences = []; foreach (string fileName in _fileNames) { OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants); @@ -638,28 +633,28 @@ public static class Program fileReferences.Add(fileInfo.Id, fileInfo); } - - Console.WriteLine("Defining agent..."); - OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = + // Define assistant + Console.WriteLine("Defining assistant..."); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ The document store contains the text of fictional stories. Always analyze the document store to provide an answer to the user's question. Never rely on your knowledge of stories not included in the document store. Always format response using markdown. """, - EnableFileSearch = true, - VectorStoreId = storeId, - }, - new Kernel()); + enableFileSearch: true, + vectorStoreId: storeId); + // Create agent + OpenAIAssistantAgent agent = new(assistant, assistantClient); + + // Create the conversation thread Console.WriteLine("Creating thread..."); - string threadId = await agent.CreateThreadAsync(); + AssistantAgentThread agentThread = new(); Console.WriteLine("Ready!"); @@ -681,11 +676,11 @@ public static class Program break; } - await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); + var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); List footnotes = []; - await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(threadId)) + await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(message, agentThread)) { // Capture annotations for footnotes footnotes.AddRange(chunk.Items.OfType()); @@ -713,8 +708,8 @@ public static class Program Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + agentThread.DeleteAsync(), + assistantClient.DeleteAssistantAsync(assistant.Id), storeClient.DeleteVectorStoreAsync(storeId), ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key)) ]); @@ -729,8 +724,6 @@ public static class Program ::: zone pivot="programming-language-python" ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import os @@ -800,8 +793,7 @@ async def main(): definition=definition, ) - print("Creating thread...") - thread = await client.beta.threads.create() + thread: AssistantAgentThread = None try: is_complete: bool = False diff --git a/semantic-kernel/Frameworks/agent/examples/example-chat-agent.md b/semantic-kernel/Frameworks/agent/examples/example-chat-agent.md index d24a85f8..291d4ba3 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-chat-agent.md +++ b/semantic-kernel/Frameworks/agent/examples/example-chat-agent.md @@ -11,11 +11,11 @@ ms.service: semantic-kernel # How-To: `ChatCompletionAgent` > [!IMPORTANT] -> This feature is in the experimental stage. Features at this stage are still under development and subject to change before advancing to the preview or release candidate stage. +> This feature is in the experimental stage. Features at this stage are under development and subject to change before advancing to the preview or release candidate stage. ## Overview -In this sample, we will explore configuring a plugin to access _GitHub_ API and provide templatized instructions to a [`ChatCompletionAgent`](../chat-completion-agent.md) to answer questions about a _GitHub_ repository. The approach will be broken down step-by-step to high-light the key parts of the coding process. As part of the task, the agent will provide document citations within the response. +In this sample, we will explore configuring a plugin to access GitHub API and provide templatized instructions to a [`ChatCompletionAgent`](../chat-completion-agent.md) to answer questions about a GitHub repository. The approach will be broken down step-by-step to high-light the key parts of the coding process. As part of the task, the agent will provide document citations within the response. Streaming will be used to deliver the agent's responses. This will provide real-time updates as the task progresses. @@ -39,7 +39,7 @@ dotnet add package Microsoft.SemanticKernel.Connectors.AzureOpenAI dotnet add package Microsoft.SemanticKernel.Agents.Core --prerelease ``` -> If managing _NuGet_ packages in _Visual Studio_, ensure `Include prerelease` is checked. +> If managing NuGet packages in Visual Studio, ensure `Include prerelease` is checked. The project file (`.csproj`) should contain the following `PackageReference` definitions: @@ -63,7 +63,7 @@ The `Agent Framework` is experimental and requires warning suppression. This ma ``` -Additionally, copy the GitHub plug-in and models (`GitHubPlugin.cs` and `GitHubModels.cs`) from [_Semantic Kernel_ `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Plugins/GitHub). Add these files in your project folder. +Additionally, copy the GitHub plug-in and models (`GitHubPlugin.cs` and `GitHubModels.cs`) from [Semantic Kernel `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/LearnResources/Plugins/GitHub). Add these files in your project folder. ::: zone-end @@ -75,10 +75,9 @@ import os import sys from datetime import datetime -from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.agents import ChatCompletionAgent, ChatHistoryAgentThread from semantic_kernel.connectors.ai import FunctionChoiceBehavior from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion -from semantic_kernel.contents import AuthorRole, ChatHistory, ChatMessageContent from semantic_kernel.functions import KernelArguments from semantic_kernel.kernel import Kernel @@ -90,7 +89,7 @@ sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) from plugins.GithubPlugin.github import GitHubPlugin, GitHubSettings # noqa: E402 ``` -Additionally, copy the GitHub plug-in and models (`github.py`) from [_Semantic Kernel_ `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/plugins/GithubPlugin). Add these files in your project folder. +Additionally, copy the GitHub plug-in and models (`github.py`) from [Semantic Kernel `LearnResources` Project](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/learn_resources/plugins/GithubPlugin). Add these files in your project folder. ::: zone-end ::: zone pivot="programming-language-java" @@ -101,9 +100,9 @@ Additionally, copy the GitHub plug-in and models (`github.py`) from [_Semantic K ## Configuration -This sample requires configuration setting in order to connect to remote services. You will need to define settings for either _OpenAI_ or _Azure OpenAI_ and also for _GitHub_. +This sample requires configuration setting in order to connect to remote services. You will need to define settings for either OpenAI or Azure OpenAI and also for GitHub. -> Note: For information on GitHub _Personal Access Tokens_, see: [Managing your personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens). +> Note: For information on GitHub Personal Access Tokens, see: [Managing your personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens). ::: zone pivot="programming-language-csharp" @@ -200,7 +199,7 @@ The coding process for this sample involves: 1. [Setup](#setup) - Initializing settings and the plug-in. 2. [`Agent` Definition](#agent-definition) - Create the `ChatCompletionAgent` with templatized instructions and plug-in. -3. [The _Chat_ Loop](#the-chat-loop) - Write the loop that drives user / agent interaction. +3. [The Chat Loop](#the-chat-loop) - Write the loop that drives user / agent interaction. The full example code is provided in the [Final](#final) section. Refer to that section for the complete implementation. @@ -291,7 +290,7 @@ settings.function_choice_behavior = FunctionChoiceBehavior.Auto() ### Agent Definition -Finally we are ready to instantiate a `ChatCompletionAgent` with its _Instructions_, associated `Kernel`, and the default _Arguments_ and _Execution Settings_. In this case, we desire to have the any plugin functions automatically executed. +Finally we are ready to instantiate a `ChatCompletionAgent` with its Instructions, associated `Kernel`, and the default Arguments and Execution Settings. In this case, we desire to have the any plugin functions automatically executed. ::: zone pivot="programming-language-csharp" ```csharp @@ -340,8 +339,7 @@ agent = ChatCompletionAgent( The current date and time is: {{$now}}. """, arguments=KernelArguments( - settings=AzureChatPromptExecutionSettings(function_choice_behavior=FunctionChoiceBehavior.Auto()), - repository="microsoft/semantic-kernel", + settings=settings, ), ) ``` @@ -353,13 +351,13 @@ agent = ChatCompletionAgent( ::: zone-end -### The _Chat_ Loop +### The Chat Loop -At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating a `ChatHistory` object to maintain the conversation state and creating an empty loop. +At last, we are able to coordinate the interaction between the user and the `Agent`. Start by creating a `ChatHistoryAgentThread` object to maintain the conversation state and creating an empty loop. ::: zone pivot="programming-language-csharp" ```csharp -ChatHistory history = []; +ChatHistoryAgentThread agentThread = new(); bool isComplete = false; do { @@ -370,7 +368,7 @@ do ::: zone pivot="programming-language-python" ```python -history = ChatHistory() +thread: ChatHistoryAgentThread = None is_complete: bool = False while not is_complete: # processing logic here @@ -383,7 +381,7 @@ while not is_complete: ::: zone-end -Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. Valid input will be added to the `ChatHistory` as a _User_ message. +Now let's capture user input within the previous loop. In this case, empty input will be ignored and the term `EXIT` will signal that the conversation is completed. ::: zone pivot="programming-language-csharp" ```csharp @@ -400,7 +398,7 @@ if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase)) break; } -history.Add(new ChatMessageContent(AuthorRole.User, input)); +var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); ``` @@ -415,8 +413,6 @@ if not user_input: if user_input.lower() == "exit": is_complete = True break - -history.add_message(ChatMessageContent(role=AuthorRole.USER, content=user_input)) ``` ::: zone-end @@ -438,7 +434,7 @@ KernelArguments arguments = { { "now", $"{now.ToShortDateString()} {now.ToShortTimeString()}" } }; -await foreach (ChatMessageContent response in agent.InvokeAsync(history, arguments)) +await foreach (ChatMessageContent response in agent.InvokeAsync(message, agentThread, options: new() { KernelArguments = arguments })) { Console.WriteLine($"{response.Content}"); } @@ -447,14 +443,13 @@ await foreach (ChatMessageContent response in agent.InvokeAsync(history, argumen ::: zone pivot="programming-language-python" ```python -from datetime import datetime - arguments = KernelArguments( now=datetime.now().strftime("%Y-%m-%d %H:%M") ) -async for response in agent.invoke(history, arguments): +async for response in agent.invoke(messages=user_input, thread=thread, arguments=arguments): print(f"{response.content}") + thread = response.thread ``` ::: zone-end @@ -539,7 +534,7 @@ public static class Program Console.WriteLine("Ready!"); - ChatHistory history = []; + ChatHistoryAgentThread agentThread = new(); bool isComplete = false; do { @@ -556,7 +551,7 @@ public static class Program break; } - history.Add(new ChatMessageContent(AuthorRole.User, input)); + var message = new ChatMessageContent(AuthorRole.User, input); Console.WriteLine(); @@ -566,7 +561,7 @@ public static class Program { { "now", $"{now.ToShortDateString()} {now.ToShortTimeString()}" } }; - await foreach (ChatMessageContent response in agent.InvokeAsync(history, arguments)) + await foreach (ChatMessageContent response in agent.InvokeAsync(message, agentThread, options: new() { KernelArguments = arguments })) { // Display response. Console.WriteLine($"{response.Content}"); @@ -585,13 +580,10 @@ import os import sys from datetime import datetime -from semantic_kernel.agents import ChatCompletionAgent -from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior +from semantic_kernel.agents import ChatCompletionAgent, ChatHistoryAgentThread +from semantic_kernel.connectors.ai import FunctionChoiceBehavior from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion -from semantic_kernel.contents.chat_history import ChatHistory -from semantic_kernel.contents.chat_message_content import ChatMessageContent -from semantic_kernel.contents.utils.author_role import AuthorRole -from semantic_kernel.functions.kernel_arguments import KernelArguments +from semantic_kernel.functions import KernelArguments from semantic_kernel.kernel import Kernel # Adjust the sys.path so we can use the GitHubPlugin and GitHubSettings classes @@ -601,13 +593,6 @@ sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) from plugins.GithubPlugin.github import GitHubPlugin, GitHubSettings # noqa: E402 -################################################################### -# The following sample demonstrates how to create a simple, # -# ChatCompletionAgent to use a GitHub plugin to interact # -# with the GitHub API. # -################################################################### - - async def main(): kernel = Kernel() @@ -620,9 +605,11 @@ async def main(): settings.function_choice_behavior = FunctionChoiceBehavior.Auto() # Set your GitHub Personal Access Token (PAT) value here - gh_settings = GitHubSettings(token="") + gh_settings = GitHubSettings(token="") # nosec kernel.add_plugin(plugin=GitHubPlugin(gh_settings), plugin_name="GithubPlugin") + current_time = datetime.now().isoformat() + # Create the agent agent = ChatCompletionAgent( kernel=kernel, @@ -636,12 +623,12 @@ async def main(): The repository you are querying is a public repository with the following name: microsoft/semantic-kernel - The current date and time is: {{$now}}. + The current date and time is: {current_time}. """, arguments=KernelArguments(settings=settings), ) - history = ChatHistory() + thread: ChatHistoryAgentThread = None is_complete: bool = False while not is_complete: user_input = input("User:> ") @@ -652,14 +639,11 @@ async def main(): is_complete = True break - history.add_message(ChatMessageContent(role=AuthorRole.USER, content=user_input)) - - arguments = KernelArguments( - now=datetime.now().strftime("%Y-%m-%d %H:%M") - ) + arguments = KernelArguments(now=datetime.now().strftime("%Y-%m-%d %H:%M")) - async for response in agent.invoke(history=history, arguments): + async for response in agent.invoke(messages=user_input, thread=thread, arguments=arguments): print(f"{response.content}") + thread = response.thread if __name__ == "__main__": diff --git a/semantic-kernel/Frameworks/agent/index.md b/semantic-kernel/Frameworks/agent/index.md index 749207bb..800b0f04 100644 --- a/semantic-kernel/Frameworks/agent/index.md +++ b/semantic-kernel/Frameworks/agent/index.md @@ -11,12 +11,17 @@ ms.service: semantic-kernel # Semantic Kernel Agent Framework > [!IMPORTANT] -> Single-agent features, such as ChatCompletionAgent and OpenAIAssistantAgent, are in the release candidate stage. These features are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. However, agent chat patterns are still in the experimental stage. These patterns are under active development and may change significantly before advancing to the preview or release candidate stage. +> `AgentChat` patterns are in the experimental stage. These patterns are under active development and may change significantly before advancing to the preview or release candidate stage. The Semantic Kernel Agent Framework provides a platform within the Semantic Kernel eco-system that allow for the creation of AI **agents** and the ability to incorporate **agentic patterns** into any application based on the same patterns and features that exist in the core Semantic Kernel framework. ## What is an AI agent? +![Blue gradient user icon representing AI agent](../../media/agentSKdocs.png) +![Pink gradient user icon representing AI agent](../../media/agentSKdocs2.png) +![Orange gradient user icon representing AI agent](../../media/agentSKdocs3.png) +![Red-pink gradient user icon representing AI agent](../../media/agentSKdocs4.png) + An **AI agent** is a software entity designed to perform tasks autonomously or semi-autonomously by recieving input, processing information, and taking actions to achieve specific goals. Agents can send and receive messages, generating responses using a combination of models, tools, human inputs, or other customizable components. @@ -50,20 +55,20 @@ Using an agent framework for application development provides advantages that ar - **Interactive and Goal-Oriented**: If your application involves goal-driven behavior (e.g., completing tasks autonomously or interacting with users to achieve specific objectives), agent-based frameworks are a better choice. Examples include virtual assistants, game AI, and task planners. -## How do I install the _Semantic Kernel Agent Framework_? +## How do I install the Semantic Kernel Agent Framework? -Installing the _Agent Framework SDK_ is specific to the distribution channel associated with your programming language. +Installing the Agent Framework SDK is specific to the distribution channel associated with your programming language. ::: zone pivot="programming-language-csharp" For .NET SDK, several NuGet packages are available. -> Note: The core _Semantic Kernel SDK_ is required in addition to any agent packages. +> Note: The core Semantic Kernel SDK is required in addition to any agent packages. Package|Description --|-- -[Microsoft.SemanticKernel](https://www.nuget.org/packages/Microsoft.SemanticKernel)|This contains the core _Semantic Kernel_ libraries for getting started with the `Agent Framework`. This must be explicitly referenced by your application. +[Microsoft.SemanticKernel](https://www.nuget.org/packages/Microsoft.SemanticKernel)|This contains the core Semantic Kernel libraries for getting started with the `Agent Framework`. This must be explicitly referenced by your application. [Microsoft.SemanticKernel.Agents.Abstractions](https://www.nuget.org/packages/Microsoft.SemanticKernel.Agents.Abstractions)|Defines the core agent abstractions for the `Agent Framework`. Generally not required to be specified as it is included in both the `Microsoft.SemanticKernel.Agents.Core` and `Microsoft.SemanticKernel.Agents.OpenAI` packages. [Microsoft.SemanticKernel.Agents.Core](https://www.nuget.org/packages/Microsoft.SemanticKernel.Agents.Core)|Includes the [`ChatCompletionAgent`](./chat-completion-agent.md) and [`AgentGroupChat`](./agent-chat.md) classes. [Microsoft.SemanticKernel.Agents.OpenAI](https://www.nuget.org/packages/Microsoft.SemanticKernel.Agents.OpenAI)|Provides ability to use the [OpenAI Assistant API](https://platform.openai.com/docs/assistants) via the [`OpenAIAssistantAgent`](./assistant-agent.md). @@ -74,7 +79,7 @@ Package|Description Module|Description --|-- -[semantic-kernel.agents](https://pypi.org/project/semantic-kernel/)|This is the _Semantic Kernel_ library for getting started with the `Agent Framework`. This must be explicitly referenced by your application. This module contains the [`ChatCompletionAgent`](./chat-completion-agent.md) and [`AgentGroupChat`](./agent-chat.md) classes, as well as the ability to use the [OpenAI Assistant API](https://platform.openai.com/docs/assistants) via the [`OpenAIAssistantAgent` or `AzureOpenAssistant`](./assistant-agent.md). +[semantic-kernel.agents](https://pypi.org/project/semantic-kernel/)|This is the Semantic Kernel library for getting started with the `Agent Framework`. This must be explicitly referenced by your application. This module contains the [`ChatCompletionAgent`](./chat-completion-agent.md), the [`OpenAIAssistantAgent`](./assistant-agent.md), the [`AzureAIAgent`](./azure-ai-agent.md), and the [`OpenAIResponsesAgent`](./responses-agent.md), as well as [`AgentGroupChat`](./agent-chat.md) class. ::: zone-end diff --git a/semantic-kernel/Frameworks/agent/responses-agent.md b/semantic-kernel/Frameworks/agent/responses-agent.md new file mode 100644 index 00000000..bca15bb5 --- /dev/null +++ b/semantic-kernel/Frameworks/agent/responses-agent.md @@ -0,0 +1,341 @@ +--- +title: Exploring the Semantic Kernel OpenAI Responses Agent +description: An exploration of the definition, behaviors, and usage patterns for a `OpenAIResponsesAgent` +zone_pivot_groups: programming-languages +author: moonbox3 +ms.topic: tutorial +ms.author: evmattso +ms.date: 04/02/2025 +ms.service: semantic-kernel +--- +# Exploring the Semantic Kernel `OpenAIResponsesAgent` + +> [!IMPORTANT] +> This feature is in the experimental stage. Features at this stage are under development and subject to change before advancing to the preview or release candidate stage. The current `OpenAIResponsesAgent` is not supported as part of Semantic Kernel's `AgentGroupChat` patterns. Stayed tuned for updates. + +Detailed API documentation related to this discussion is available at: + +::: zone pivot="programming-language-csharp" +> The `OpenAIResponsesAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +- [`AzureResponsesAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.azure_responses_agent.azureresponsesagent) +- [`OpenAIResponsesAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.open_ai_responses_agent.openairesponsesagent) + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + +## What is a Responses Agent? + +The OpenAI Responses API is OpenAI's most advanced interface for generating model responses. It supports text and image inputs, and text outputs. You are able to create stateful interactions with the model, using the output of previous responses as input. It is also possible to extend the model's capabilities with built-in tools for file search, web search, computer use, and more. + +- [OpenAI Responses API](https://platform.openai.com/docs/api-reference/responses) +- [Responses API in Azure](/azure/ai-services/openai/how-to/responses?tabs=python-secure) + + +## Preparing Your Development Environment + +To proceed with developing an `OpenAIResponsesAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" +> The `OpenAIResponsesAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package: + +```bash +pip install semantic-kernel +``` + +> [!Important] +> The `OpenAIResponsesAgent` is supported in Semantic Kernel Python packages 1.27.0 and later. + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + +## Creating an `OpenAIResponsesAgent` + +Creating an `OpenAIResponsesAgent` requires first creating a client to be able to talk a remote service. + +::: zone pivot="programming-language-csharp" +> The `OpenAIResponsesAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +To configure the model used by the OpenAI or Azure OpenAI Responses API, a new environment variable is introduced: + +```bash +OPENAI_RESPONSES_MODEL_ID="" +AZURE_OPENAI_RESPONSES_MODEL_ID="" +``` + +Set the appropriate variable depending on which provider you're using. + +> [!TIP] +> The minimum allowed Azure OpenAI api version is `2025-03-01-preview`. Please visit the following [link](azure/ai-services/openai/how-to/responses) to view region availability, model support, and further details. + +To create an `AzureResponsesAgent` to use with Azure OpenAI models: + +```python +from semantic_kernel.agents import AzureResponsesAgent + +# Set up the client and model using Azure OpenAI Resources +client, model = AzureResponsesAgent.setup_resources() + +# Create the AzureResponsesAgent instance using the client and the model +agent = AzureResponsesAgent( + ai_model_id=model, + client=client, + instructions="your instructions", + name="name", +) +``` + +Alternatively, to create an `OpenAIResponsesAgent` to use with OpenAI models: + +```python +from semantic_kernel.agents import OpenAIResponsesAgent + +# Set up the client and model using OpenAI Resources +client, model = OpenAIResponsesAgent.setup_resources() + +# Create the OpenAIResponsesAgent instance using the client and the model +agent = OpenAIResponsesAgent( + ai_model_id=model, + client=client, + instructions="your instructions", + name="name", +) +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Using an `OpenAIResponsesAgent` + +::: zone pivot="programming-language-csharp" +> The `OpenAIResponsesAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" +The OpenAI Responses API supports optional remote storage of conversations. By default, when using a `ResponsesAgentThread`, responses are stored remotely. This enables the use of the Responses API's `previous_response_id` for maintaining context across invocations. + +Each conversation is treated as a thread, identified by a unique string ID. All interactions with your `OpenAIResponsesAgent` are scoped to this thread identifier. + +The underlying mechanics of the Responses API thread are abstracted by the `ResponsesAgentThread` class, which implements the `AgentThread` interface. + +The `OpenAIResponsesAgent` currently only supports threads of type `ResponsesAgentThread`. + +You can invoke the `OpenAIResponsesAgent` without specifying an `AgentThread`, to start a new thread and a new `AgentThread` will be returned as part of the response. + +```python +from semantic_kernel.agents import AzureResponsesAgent + +# Set up the client and model using Azure OpenAI Resources +client, model = AzureResponsesAgent.setup_resources() + +# Create the AzureResponsesAgent instance using the client and the model +agent = AzureResponsesAgent( + ai_model_id=model, + client=client, + instructions="your instructions", + name="name", +) + +USER_INPUTS = [ + "My name is John Doe.", + "Tell me a joke", + "Explain why this is funny.", + "What have we been talking about?", +] + +thread = None + +# Generate the agent response(s) +for user_input in USER_INPUTS: + print(f"# User: '{user_input}'") + # Invoke the agent for the current message and print the response + response = await agent.get_response(messages=user_input, thread=thread) + print(f"# {response.name}: {response.content}") + # Update the thread so the previous response id is used + thread = response.thread + +# Delete the thread when it is no longer needed +await thread.delete() if thread else None +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Handling Intermediate Messages with an `OpenAIResponsesAgent` + +The Semantic Kernel `OpenAIResponsesAgent` is designed to invoke an agent that fulfills user queries or questions. During invocation, the agent may execute tools to derive the final answer. To access intermediate messages produced during this process, callers can supply a callback function that handles instances of `FunctionCallContent` or `FunctionResultContent`. + +::: zone pivot="programming-language-csharp" +> The `OpenAIResponsesAgent` is coming soon. +::: zone-end + +::: zone pivot="programming-language-python" + +Configuring the `on_intermediate_message` callback within `agent.invoke(...)` or `agent.invoke_stream(...)` allows the caller to receive intermediate messages generated during the process of formulating the agent's final response. + +```python +import asyncio +from typing import Annotated + +from semantic_kernel.agents import AzureResponsesAgent +from semantic_kernel.contents import AuthorRole, ChatMessageContent, FunctionCallContent, FunctionResultContent +from semantic_kernel.functions import kernel_function + +class MenuPlugin: + """A sample Menu Plugin used for the concept sample.""" + + @kernel_function(description="Provides a list of specials from the menu.") + def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]: + return """ + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + """ + + @kernel_function(description="Provides the price of the requested menu item.") + def get_item_price( + self, menu_item: Annotated[str, "The name of the menu item."] + ) -> Annotated[str, "Returns the price of the menu item."]: + return "$9.99" + +# Define a list to hold callback message content +intermediate_steps: list[ChatMessageContent] = [] + +# Define an async method to handle the `on_intermediate_message` callback +async def handle_intermediate_steps(message: ChatMessageContent) -> None: + intermediate_steps.append(message) + +async def main(): + # Set up the client and model using Azure OpenAI Resources + client, model = AzureResponsesAgent.setup_resources() + + # Create the AzureResponsesAgent instance using the client and model + agent = AzureResponsesAgent( + ai_model_id=model, + client=client, + instructions="your instructions", + name="name", + plugins=[MenuPlugin()], + ) + + user_inputs = [ + "Hello", + "What is the special soup?", + "What is the special drink?", + "How much is that?", + "Thank you", + ] + + thread = None + + # Generate the agent response(s) + for user_input in user_inputs: + print(f"# {AuthorRole.USER}: '{user_input}'") + async for response in agent.invoke( + messages=user_input, + thread=thread, + on_intermediate_message=handle_intermediate_steps, + ): + thread = response.thread + print(f"# {response.name}: {response.content}") + + # Delete the thread when it is no longer needed + await thread.delete() if thread else None + + # Print the intermediate steps + print("\nIntermediate Steps:") + for msg in intermediate_steps: + if any(isinstance(item, FunctionResultContent) for item in msg.items): + for fr in msg.items: + if isinstance(fr, FunctionResultContent): + print(f"Function Result:> {fr.result} for function: {fr.name}") + elif any(isinstance(item, FunctionCallContent) for item in msg.items): + for fcc in msg.items: + if isinstance(fcc, FunctionCallContent): + print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}") + else: + print(f"{msg.role}: {msg.content}") + +if __name__ == "__main__": + asyncio.run(main()) +``` + +The following demonstrates sample output from the agent invocation process: + +```bash +Sample Output: + +# AuthorRole.USER: 'Hello' +# Host: Hi there! How can I assist you with the menu today? +# AuthorRole.USER: 'What is the special soup?' +# Host: The special soup is Clam Chowder. +# AuthorRole.USER: 'What is the special drink?' +# Host: The special drink is Chai Tea. +# AuthorRole.USER: 'How much is that?' +# Host: Could you please specify the menu item you are asking about? +# AuthorRole.USER: 'Thank you' +# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. + +Intermediate Steps: +AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today? +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special soup is Clam Chowder. +AuthorRole.ASSISTANT: +Function Result:> + Special Soup: Clam Chowder + Special Salad: Cobb Salad + Special Drink: Chai Tea + for function: MenuPlugin-get_specials +AuthorRole.ASSISTANT: The special drink is Chai Tea. +AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about? +AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask. +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +> [!div class="nextstepaction"] +> [Explore Agent Collaboration in Agent Chat](./agent-chat.md) + diff --git a/semantic-kernel/concepts/ai-services/TOC.yml b/semantic-kernel/concepts/ai-services/TOC.yml index 8d24a54b..fd35b379 100644 --- a/semantic-kernel/concepts/ai-services/TOC.yml +++ b/semantic-kernel/concepts/ai-services/TOC.yml @@ -6,4 +6,6 @@ - name: Embedding generation href: embedding-generation/TOC.yml - name: AI Integrations - href: integrations.md \ No newline at end of file + href: integrations.md +- name: Realtime + href: realtime.md \ No newline at end of file diff --git a/semantic-kernel/concepts/ai-services/index.md b/semantic-kernel/concepts/ai-services/index.md index 3e473aec..9ce5d2fd 100644 --- a/semantic-kernel/concepts/ai-services/index.md +++ b/semantic-kernel/concepts/ai-services/index.md @@ -14,21 +14,23 @@ One of the main features of Semantic Kernel is its ability to add different AI s Within Semantic Kernel, there are interfaces for the most popular AI tasks. In the table below, you can see the services that are supported by each of the SDKs. -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | -| Text generation | ✅ | ✅ | ✅ | -| Embedding generation (Experimental) | ✅ | ✅ | ✅ | -| Text-to-image (Experimental) | ✅ | ✅ | ❌ | -| Image-to-text (Experimental) | ✅ | ❌ | ❌ | -| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | -| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| Services | C# | Python | Java | Notes | +| --------------------------------------------- | :---: | :----: | :---: | ----- | +| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | +| Text generation | ✅ | ✅ | ✅ | +| Embedding generation (Experimental) | ✅ | ✅ | ✅ | +| Text-to-image (Experimental) | ✅ | ✅ | ❌ | +| Image-to-text (Experimental) | ✅ | ❌ | ❌ | +| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | +| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| [Realtime](./realtime.md) (Experimental) | ❌ | ✅ | ❌ | > [!TIP] > In most scenarios, you will only need to add chat completion to your kernel, but to support multi-modal AI, you can add any of the above services to your kernel. ## Next steps + To learn more about each of the services, please refer to the specific articles for each service type. In each of the articles we provide sample code for adding the service to the kernel across multiple AI service providers. > [!div class="nextstepaction"] -> [Learn about chat completion](./chat-completion/index.md) \ No newline at end of file +> [Learn about chat completion](./chat-completion/index.md) diff --git a/semantic-kernel/concepts/ai-services/integrations.md b/semantic-kernel/concepts/ai-services/integrations.md index a8d34ec5..d30a6984 100644 --- a/semantic-kernel/concepts/ai-services/integrations.md +++ b/semantic-kernel/concepts/ai-services/integrations.md @@ -18,21 +18,22 @@ With the available AI connectors, developers can easily build AI agents with swa ### AI Services -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | -| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | -| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | -| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | -| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | -| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | -| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Services | C# | Python | Java | Notes | +| ------------------------------ | :---: | :----: | :---: | -------------------------------- | +| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | +| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | +| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | +| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | +| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | +| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | +| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Realtime (Experimental) | ❌ | ✅ | ❌ | Example: gpt-4o-realtime-preview | ## Additional plugins If you want to extend the functionality of your AI agent, you can use plugins to integrate with other Microsoft services. Here are some of the plugins that are available for Semantic Kernel: -| Plugin | C# | Python | Java | Description | -| ---------- | :-: | :----: | :--: | ----------- | -| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | -| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | +| Plugin | C# | Python | Java | Description | +| ------------------------------------- | :---: | :----: | :---: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | +| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | diff --git a/semantic-kernel/concepts/ai-services/realtime.md b/semantic-kernel/concepts/ai-services/realtime.md new file mode 100644 index 00000000..dd559e91 --- /dev/null +++ b/semantic-kernel/concepts/ai-services/realtime.md @@ -0,0 +1,189 @@ +--- +title: Realtime AI Integrations for Semantic Kernel +description: Learn about realtime multi-modal AI integrations available in Semantic Kernel. +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Realtime Multi-modal APIs + +The first realtime API integration for Semantic Kernel has been added, it is currently only available in Python and considered experimental. This is because the underlying services are still being developed and are subject to changes and we might need to make breaking changes to the API in Semantic Kernel as we learn from customers how to use this and as we add other providers of these kinds of models and APIs. + +## Realtime Client abstraction + +To support different realtime APIs from different vendors, using different protocols, a new client abstraction has been added to the kernel. This client is used to connect to the realtime service and send and receive messages. +The client is responsible for handling the connection to the service, sending messages, and receiving messages. The client is also responsible for handling any errors that occur during the connection or message sending/receiving process. Considering the way these models work, they can be considered agents more than regular chat completions, therefore they also take instructions, rather than a system message, they keep their own internal state and can be invoked to do work on our behalf. +### Realtime API + +Any realtime client implements the following methods: + +| Method | Description | +| ---------------- | ------------------------------------------------------------------------------------------------------------------ | +| `create_session` | Creates a new session | +| `update_session` | Updates an existing session | +| `delete_session` | Deletes an existing session | +| `receive` | This is a asynchronous generator method that listens for messages from the service and yields them as they arrive. | +| `send` | Sends a message to the service | + +### Python implementations + +The python version of Semantic Kernel currently supports the following realtime clients: + +| Client | Protocol | Modalities | Function calling enabled | Description | +| ------ | --------- | ------------ | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| OpenAI | Websocket | Text & Audio | Yes | The OpenAI Realtime API is a websocket based api that allows you to send and receive messages in realtime, this connector uses the OpenAI Python package to connect and receive and send messages. | +| OpenAI | WebRTC | Text & Audio | Yes | The OpenAI Realtime API is a WebRTC based api that allows you to send and receive messages in realtime, it needs a webRTC compatible audio track at session creation time. | +| Azure | Websocket | Text & Audio | Yes | The Azure Realtime API is a websocket based api that allows you to send and receive messages in realtime, this uses the same package as the OpenAI websocket connector. | + +## Getting started + +To get started with the Realtime API, you need to install the `semantic-kernel` package with the `realtime` extra. + +```bash +pip install semantic-kernel[realtime] +``` + +Depending on how you want to handle audio, you might need additional packages to interface with speakers and microphones, like `pyaudio` or `sounddevice`. + +### Websocket clients + +Then you can create a kernel and add the realtime client to it, this shows how to do that with a AzureRealtimeWebsocket connection, you can replace AzureRealtimeWebsocket with OpenAIRealtimeWebsocket without any further changes. + +```python +from semantic_kernel.connectors.ai.open_ai import ( + AzureRealtimeWebsocket, + AzureRealtimeExecutionSettings, + ListenEvents, +) +from semantic_kernel.contents import RealtimeAudioEvent, RealtimeTextEvent + +# this will use environment variables to get the api key, endpoint, api version and deployment name. +realtime_client = AzureRealtimeWebsocket() +settings = AzureRealtimeExecutionSettings(voice='alloy') +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event: + # receiving a piece of audio (and send it to a undefined audio player) + case RealtimeAudioEvent(): + await audio_player.add_audio(event.audio) + # receiving a piece of audio transcript + case RealtimeTextEvent(): + # Semantic Kernel parses the transcript to a TextContent object captured in a RealtimeTextEvent + print(event.text.text, end="") + case _: + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +There are two important things to note, the first is that the `realtime_client` is an async context manager, this means that you can use it in an async function and use `async with` to create the session. +The second is that the `receive` method is an async generator, this means that you can use it in a for loop to receive messages as they arrive. + +### WebRTC client + +The setup of a WebRTC connection is a bit more complex and so we need a extra parameter when creating the client. This parameter, `audio_track` needs to be a object that implements the `MediaStreamTrack` protocol of the `aiortc` package, this is also demonstrated in the samples that are linked below. + +To create a client that uses WebRTC, you would do the following: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event.event_type: + # receiving a piece of audio (and send it to a undefined audio player) + case "audio": + await audio_player.add_audio(event.audio) + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +Both of these samples receive the audio as RealtimeAudioEvent and then they pass that to a unspecified audio_player object. + +### Audio output callback + +Next to this we have a parameter called `audio_output_callback` on the `receive` method and on the class creation. This callback will be called first before any further handling of the audio and gets a `numpy` array of the audio data, instead of it being parsed into AudioContent and returned as a RealtimeAudioEvent that you can then handle, which is what happens above. This has shown to give smoother audio output because there is less overhead between the audio data coming in and it being given to the player. + +This example shows how to define and use the `audio_output_callback`: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +class AudioPlayer: + async def play_audio(self, content: np.ndarray): + # implement the audio player + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(audio_output_callback=audio_player.play_audio): + match event.event_type: + # no need to handle case: "audio" + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +### Samples + +There are four samples in [our repo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/realtime), they cover both the basics using both websockets and WebRTC, as well as a more complex setup including function calling. Finally there is a more [complex demo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/demos/call_automation) that uses [Azure Communication Services](/azure/communication-services/) to allow you to call your Semantic Kernel enhanced realtime API. diff --git a/semantic-kernel/concepts/plugins/adding-native-plugins.md b/semantic-kernel/concepts/plugins/adding-native-plugins.md index 68d3dcba..f2f8be4e 100644 --- a/semantic-kernel/concepts/plugins/adding-native-plugins.md +++ b/semantic-kernel/concepts/plugins/adding-native-plugins.md @@ -18,6 +18,7 @@ Behind the scenes, Semantic Kernel will then use the descriptions you provide, a ## Providing the LLM with the right information When authoring a plugin, you need to provide the AI agent with the right information to understand the capabilities of the plugin and its functions. This includes: + - The name of the plugin - The names of the functions - The descriptions of the functions @@ -34,6 +35,7 @@ Below, we'll walk through the two different ways of providing your AI agent with The easiest way to create a native plugin is to start with a class and then add methods annotated with the `KernelFunction` attribute. It is also recommended to liberally use the `Description` annotation to provide the AI agent with the necessary information to understand the function. ::: zone pivot="programming-language-csharp" + ```csharp public class LightsPlugin { @@ -73,9 +75,11 @@ public class LightsPlugin } } ``` + ::: zone-end ::: zone pivot="programming-language-python" + ```python from typing import List, Optional, Annotated @@ -102,6 +106,7 @@ class LightsPlugin: return light return None ``` + ::: zone-end ::: zone pivot="programming-language-java" @@ -110,10 +115,10 @@ class LightsPlugin: ::: zone-end +::: zone pivot="programming-language-csharp" > [!TIP] > Because the LLMs are predominantly trained on Python code, it is recommended to use snake_case for function names and parameters (even if you're using C# or Java). This will help the AI agent better understand the function and its parameters. -::: zone pivot="programming-language-csharp" > [!TIP] > Your functions can specify `Kernel`, `KernelArguments`, `ILoggerFactory`, `ILogger`, `IAIServiceSelector`, `CultureInfo`, `IFormatProvider`, `CancellationToken` as parameters and these will not be advertised to the LLM and will be automatically set when the function is called. > If you rely on `KernelArguments` instead of explicit input arguments then your code will be responsible for performing type conversions. @@ -122,6 +127,7 @@ class LightsPlugin: If your function has a complex object as an input variable, Semantic Kernel will also generate a schema for that object and pass it to the AI agent. Similar to functions, you should provide `Description` annotations for properties that are non-obvious to the AI. Below is the definition for the `LightState` class and the `Brightness` enum. ::: zone pivot="programming-language-csharp" + ```csharp using System.Text.Json.Serialization; @@ -152,19 +158,28 @@ public enum Brightness High } ``` + ::: zone-end ::: zone pivot="programming-language-python" + ```python +from enum import Enum from typing import TypedDict +class Brightness(Enum): + LOW = "low" + MEDIUM = "medium" + HIGH = "high" + class LightModel(TypedDict): id: int name: str is_on: bool | None - brightness: int | None - hex: str | None + brightness: Brightness | None + color: Annotated[str | None, "The color of the light with a hex code (ensure you include the # symbol)"] ``` + ::: zone-end ::: zone pivot="programming-language-java" @@ -176,12 +191,29 @@ class LightModel(TypedDict): > [!NOTE] > While this is a "fun" example, it does a good job showing just how complex a plugin's parameters can be. In this single case, we have a complex object with _four_ different types of properties: an integer, string, boolean, and enum. Semantic Kernel's value is that it can automatically generate the schema for this object and pass it to the AI agent and marshal the parameters generated by the AI agent into the correct object. +::: zone pivot="programming-language-csharp" + Once you're done authoring your plugin class, you can add it to the kernel using the `AddFromType<>` or `AddFromObject` methods. +::: zone-end + +::: zone pivot="programming-language-python" + +Once you're done authoring your plugin class, you can add it to the kernel using the `add_plugin` method. + +::: zone-end + +::: zone pivot="programming-language-java" + +Once you're done authoring your plugin class, you can add it to the kernel using the `AddFromType<>` or `AddFromObject` methods. + +::: zone-end + > [!TIP] > When creating a function, always ask yourself "how can I give the AI additional help to use this function?" This can include using specific input types (avoid strings where possible), providing descriptions, and examples. ::: zone pivot="programming-language-csharp" + #### Adding a plugin using the `AddFromObject` method The `AddFromObject` method allows you to add an instance of the plugin class directly to the plugin collection in case you want to directly control how the plugin is constructed. @@ -325,9 +357,11 @@ builder.Services.AddTransient((serviceProvider)=> { return new Kernel(serviceProvider, pluginCollection); }); ``` + ::: zone-end ::: zone pivot="programming-language-python" + #### Adding a plugin using the `add_plugin` method The `add_plugin` method allows you to add a plugin instance to the kernel. Below is an example of how you can construct the `LightsPlugin` class and add it to the kernel. @@ -349,8 +383,8 @@ lights_plugin = LightsPlugin(lights) # Add the plugin to the kernel kernel.add_plugin(lights_plugin) ``` -::: zone-end +::: zone-end ::: zone pivot="programming-language-java" @@ -497,7 +531,47 @@ This approach eliminates the need to manually provide and update the return type ::: zone-end +::: zone pivot="programming-language-python" + +### Providing more details about the functions + +When creating a plugin in Python, you can provide additional information about the functions in the `kernel_function` decorator. This information will be used by the AI agent to understand the functions better. + +```python +from typing import List, Optional, Annotated + +class LightsPlugin: + def __init__(self, lights: List[LightModel]): + self._lights = lights + + @kernel_function(name="GetLights", description="Gets a list of lights and their current state") + async def get_lights(self) -> List[LightModel]: + """Gets a list of lights and their current state.""" + return self._lights + + @kernel_function(name="ChangeState", description="Changes the state of the light") + async def change_state( + self, + change_state: LightModel + ) -> Optional[LightModel]: + """Changes the state of the light.""" + for light in self._lights: + if light["id"] == change_state["id"]: + light["is_on"] = change_state.get("is_on", light["is_on"]) + light["brightness"] = change_state.get("brightness", light["brightness"]) + light["hex"] = change_state.get("hex", light["hex"]) + return light + return None +``` + +The sample above shows how to override the function name and provide a description for the function. By default, the function name is the name of the function and the description is empty. If the function name is descriptive enough, you won't need a description, which will save you tokens. However, if the function behavior is not obvious from the name, you should provide a description for the AI. + +Because the LLMs are predominantly trained on Python code, it is recommended to use function names that follow the [Python naming conventions](https://peps.python.org/pep-0008/#function-and-variable-names), which means you rarely need to override the function names if you follow the conventions in your Python code. + +::: zone-end + ## Next steps + Now that you know how to create a plugin, you can now learn how to use them with your AI agent. Depending on the type of functions you've added to your plugins, there are different patterns you should follow. For retrieval functions, refer to the [using retrieval functions](./using-data-retrieval-functions-for-rag.md) article. For task automation functions, refer to the [using task automation functions](./using-task-automation-functions.md) article. > [!div class="nextstepaction"] diff --git a/semantic-kernel/concepts/plugins/adding-openapi-plugins.md b/semantic-kernel/concepts/plugins/adding-openapi-plugins.md index f9f2b402..4f4560ab 100644 --- a/semantic-kernel/concepts/plugins/adding-openapi-plugins.md +++ b/semantic-kernel/concepts/plugins/adding-openapi-plugins.md @@ -345,6 +345,18 @@ To handle payloads with non-unique property names, consider the following altern If payloads schemas use any of the `oneOf`, `anyOf`, `allOf` composite keywords or recursive references, consider disabling dynamic payload construction and allow the LLM to create the payload based on its schema, as explained in the [The payload parameter](./adding-openapi-plugins.md#the-payload-parameter) section. +#### Note on the `oneOf` and `anyOf` Keywords +The `anyOf` and `oneOf` keywords assume that a payload can be composed of properties defined by multiple schemas. +The `anyOf` keyword allows a payload to include properties defined in one or more schemas, while `oneOf` restricts the payload to contain properties from only one schema among the many provided. +For more information, you can refer to the [Swagger documentation on oneOf and anyOf](https://swagger.io/docs/specification/v3_0/data-models/oneof-anyof-allof-not/). + +With both `anyOf` and `oneOf` keywords, which offer alternatives to the payload structure, it's impossible to predict which alternative a caller will choose +when invoking operations that define payloads with these keywords. For example, it is not possible to determine in advance whether a caller will invoke an operation with a Dog or Cat object, or with an object composed of some or perhaps all properties from the PetByAge and PetByType schemas +described in the examples for `anyOf` and `oneOf` in the [Swagger documentation](https://swagger.io/docs/specification/v3_0/data-models/oneof-anyof-allof-not/). +As a result, because there's no set of parameters known in advance that Semantic Kernel can use to create the a plugin function with for such operations, Semantic Kernel creates a function with only one [payload](./adding-openapi-plugins.md#the-payload-parameter) parameter +having a schema from the operation describing a multitude of possible alternatives, offloading the payload creation to the operation caller: LLM or calling code +that must have all the context to know which one of the available alternatives to invoke the function with. + ### Payload namespacing Payload namespacing helps prevent naming conflicts that can occur due to non-unique property names in OpenAPI plugin payloads. diff --git a/semantic-kernel/concepts/vector-store-connectors/TOC.yml b/semantic-kernel/concepts/vector-store-connectors/TOC.yml index b7cfa81b..85f04702 100644 --- a/semantic-kernel/concepts/vector-store-connectors/TOC.yml +++ b/semantic-kernel/concepts/vector-store-connectors/TOC.yml @@ -12,6 +12,8 @@ href: embedding-generation.md - name: Vector Search href: vector-search.md +- name: Hybrid Search + href: hybrid-search.md - name: Serialization of data models href: serialization.md - name: Legacy Memory Stores diff --git a/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md b/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md index 736d5d59..f3769bec 100644 --- a/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md +++ b/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md @@ -99,11 +99,12 @@ public async Task GenerateEmbeddingsAndSearchAsync( await textEmbeddingGenerationService.GenerateEmbeddingAsync(descriptionText); // Search using the already generated embedding. - List> searchResult = await collection.VectorizedSearchAsync(searchEmbedding).ToListAsync(); + VectorSearchResults searchResult = await collection.VectorizedSearchAsync(searchEmbedding); + List> resultItems = await searchResult.Results.ToListAsync(); // Print the first search result. - Console.WriteLine("Score for first result: " + searchResult.FirstOrDefault()?.Score); - Console.WriteLine("Hotel description for first result: " + searchResult.FirstOrDefault()?.Record.Description); + Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); + Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); } ``` diff --git a/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md b/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md index fd95fc28..9f6faeaf 100644 --- a/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md @@ -107,26 +107,31 @@ There may be cases where the database doesn't support excluding vectors in which returning them is acceptable. 1.9 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should also -respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. +respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. 1.10 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should simulate -the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database +the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database does not support this natively. To simulate this behavior, the implementation should fetch a number of results equal to Top + Skip, and then skip the first Skip number of results before returning the remaining results. 1.11 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should ignore -the `IncludeTotalCount` option provided via `VectorSearchOptions` if the database +the `IncludeTotalCount` option provided via `VectorSearchOptions` if the database does not support this natively. -1.12 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should default -to the first vector if the `VectorPropertyName` option was not provided via `VectorSearchOptions`. -If a user does provide this value, the expected name should be the property name from the data model -and not any customized name that the property may be stored under in the database. E.g. let's say -the user has a data model property called `TextEmbedding` and they decorated the property with a -`JsonPropertyNameAttribute` that indicates that it should be serialized as `text_embedding`. Assuming -that the database is json based, it means that the property should be stored in the database with the -name `text_embedding`. When specifying the `VectorPropertyName` option, the user should always provide +1.12 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should not require +`VectorPropertyName` or `VectorProperty` to be specified if only one vector exists on the data model. +In this case that single vector should automatically become the search target. If no vector or +multiple vectors exists on the data model, and no `VectorPropertyName` or `VectorProperty` is provided +the search method should throw. + +When using `VectorPropertyName`, if a user does provide this value, the expected name should be the +property name from the data model and not any customized name that the property may be stored under +in the database. E.g. let's say the user has a data model property called `TextEmbedding` and they +decorated the property with a `JsonPropertyNameAttribute` that indicates that it should be serialized +as `text_embedding`. Assuming that the database is json based, it means that the property should be +stored in the database with the name `text_embedding`. + When specifying the `VectorPropertyName` option, the user should always provide `TextEmbedding` as the value. This is to ensure that where different connectors are used with the same data model, the user can always use the same property names, even though the storage name of the property may be different. @@ -299,7 +304,12 @@ To support this scenario, the connector must fulfil the following requirements: - Use the `VectorStoreRecordDefinition` to create collections / indexes. - Avoid doing reflection on the data model if a custom mapper and `VectorStoreRecordDefinition` is supplied -### 10. Support Vector Store Record Collection factory +### 10. Support Vector Store Record Collection factory (Deprecated) + +> [!IMPORTANT] +> Support for Vector Store Record Collection factories is now deprecated. The recommended pattern is to unseal +> the VectorStore class and make the `GetCollection` method virtual so that it can be overridden by developers +> who require custom construction of collections. The `IVectorStore.GetCollection` method can be used to create instances of `IVectorStoreRecordCollection`. Some connectors however may allow or require users to provide additional configuration options @@ -404,7 +414,7 @@ into small enough batches already so that parallel requests will succeed without E.g. here is an example where batching is simulated with requests happening in parallel. ```csharp -public Task DeleteBatchAsync(IEnumerable keys, DeleteRecordOptions? options = default, CancellationToken cancellationToken = default) +public Task DeleteBatchAsync(IEnumerable keys, CancellationToken cancellationToken = default) { if (keys == null) { @@ -412,7 +422,7 @@ public Task DeleteBatchAsync(IEnumerable keys, DeleteRecordOptions? opti } // Remove records in parallel. - var tasks = keys.Select(key => this.DeleteAsync(key, options, cancellationToken)); + var tasks = keys.Select(key => this.DeleteAsync(key, cancellationToken)); return Task.WhenAll(tasks); } ``` @@ -422,13 +432,14 @@ client so that it may send the entire set in one request. ## Recommended common patterns and pratices +1. Keep `IVectorStore` and `IVectorStoreRecordCollection` implementations unsealed with virtual methods, so that developers can inherit and override if needed. 1. Always use options classes for optional settings with smart defaults. 1. Keep required parameters on the main signature and move optional parameters to options. Here is an example of an `IVectorStoreRecordCollection` constructor following this pattern. ```csharp -public sealed class MyDBVectorStoreRecordCollection : IVectorStoreRecordCollection +public class MyDBVectorStoreRecordCollection : IVectorStoreRecordCollection { public MyDBVectorStoreRecordCollection(MyDBClient myDBClient, string collectionName, MyDBVectorStoreRecordCollectionOptions? options = default) { @@ -437,13 +448,18 @@ public sealed class MyDBVectorStoreRecordCollection : IVectorStoreRecor ... } -public sealed class MyDBVectorStoreRecordCollectionOptions +public class MyDBVectorStoreRecordCollectionOptions { public VectorStoreRecordDefinition? VectorStoreRecordDefinition { get; init; } = null; public IVectorStoreRecordMapper? MyDbRecordCustomMapper { get; init; } = null; } ``` +## SDK Changes + +Please also see the following articles for a history of changes to the SDK and therefore implementation requirements: + +1. [Vector Store Changes March 2025](../../../support/migration/vectorstore-march-2025.md) ::: zone-end ::: zone pivot="programming-language-python" diff --git a/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md b/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md index 81deabad..8cd2a2d5 100644 --- a/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md +++ b/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md @@ -12,6 +12,10 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. +> [!WARNING] +> Support for custom mappers may be deprecated in future, since filtering and target property selection is not able to target the top level mapped types. + +::: zone pivot="programming-language-csharp" In this how to, we will show how you can replace the default mapper for a vector store record collection with your own mapper. @@ -37,19 +41,11 @@ All Vector Store connector implementations allow you to provide a custom mapper. The underlying data stores of each Vector Store connector have different ways of storing data. Therefore what you are mapping to on the storage side may differ for each connector. -::: zone pivot="programming-language-csharp" E.g. if using the Qdrant connector, the storage type is a `PointStruct` class provided by the Qdrant SDK. If using the Redis JSON connector, the storage type is a `string` key and a `JsonNode`, while if using a JSON HashSet connector, the storage type is a `string` key and a `HashEntry` array. -::: zone-end -::: zone pivot="programming-language-python" -::: zone-end -::: zone pivot="programming-language-java" -::: zone-end If you want to do custom mapping, and you want to use multiple connector types, you will therefore need to implement a mapper for each connector type. -::: zone pivot="programming-language-csharp" - ## Creating the data model Our first step is to create a data model. In this case we will not annotate the data model with attributes, since we will provide a separate record definition @@ -204,18 +200,15 @@ When using `IVectorStore` to get `IVectorStoreRecordCollection` object instances the `GetCollection` method. This is because custom mappers differ for each Vector Store type, and would make it impossible to use `IVectorStore` to communicate with any vector store implementation. -It is however possible to provide a factory when constructing a Vector Store implementation. This can be used to customize `IVectorStoreRecordCollection` -instances as they are created. +It is however possible to override the default implementation of `GetCollection` and provide your own custom implementation of the vector store. -Here is an example of such a factory, which checks if `CreateCollection` was called with the product definition and data type, and if so -injects the custom mapper and switches on named vectors mode. +Here is an example where we inherit from the `QdrantVectorStore` and override the `GetCollection` method to do the custom construction. ```csharp -public class QdrantCollectionFactory(VectorStoreRecordDefinition productDefinition) : IQdrantVectorStoreRecordCollectionFactory +private sealed class QdrantCustomVectorStore(QdrantClient qdrantClient, VectorStoreRecordDefinition productDefinition) + : QdrantVectorStore(qdrantClient) { - public IVectorStoreRecordCollection CreateVectorStoreRecordCollection(QdrantClient qdrantClient, string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition) - where TKey : notnull - where TRecord : class + public override IVectorStoreRecordCollection GetCollection(string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition = null) { // If the record definition is the product definition and the record type is the product data // model, inject the custom mapper into the collection options. @@ -233,39 +226,30 @@ public class QdrantCollectionFactory(VectorStoreRecordDefinition productDefiniti return customCollection!; } - // Otherwise, just create a standard collection with the default mapper. - var collection = new QdrantVectorStoreRecordCollection( - qdrantClient, - name, - new() - { - VectorStoreRecordDefinition = vectorStoreRecordDefinition - }) as IVectorStoreRecordCollection; - return collection!; + // Otherwise, just create a standard collection. + return base.GetCollection(name, vectorStoreRecordDefinition); } } ``` -To use the collection factory, pass it to the Vector Store when constructing it, or when registering it with the dependency injection container. +To use the replacement vector store, register it with your dependency injection container or use just use it directly as you would a regular `QdrantVectorStore`. ```csharp // When registering with the dependency injection container on the kernel builder. -kernelBuilder.AddQdrantVectorStore( - "localhost", - options: new() +kernelBuilder.Services.AddTransient( + (sp) => { - VectorStoreCollectionFactory = new QdrantCollectionFactory(productDefinition) + return new QdrantCustomVectorStore( + new QdrantClient("localhost"), + productDefinition); }); ``` ```csharp // When constructing the Vector Store instance directly. -var vectorStore = new QdrantVectorStore( +var vectorStore = new QdrantCustomVectorStore( new QdrantClient("localhost"), - new() - { - VectorStoreCollectionFactory = new QdrantCollectionFactory(productDefinition) - }); + productDefinition); ``` Now you can use the vector store as normal to get a collection. diff --git a/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md b/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md new file mode 100644 index 00000000..f962d91c --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md @@ -0,0 +1,270 @@ +--- +title: Hybrid search using Semantic Kernel Vector Store connectors (Preview) +description: Describes the different options you can use when doing a hybrid search using Semantic Kernel vector store connectors. +zone_pivot_groups: programming-languages +author: westey-m +ms.topic: conceptual +ms.author: westey +ms.date: 03/06/2025 +ms.service: semantic-kernel +--- +# Hybrid search using Semantic Kernel Vector Store connectors (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +Semantic Kernel provides hybrid search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. + +Currently the type of hybrid search supported is based on a vector search, plus a keyword search, both of which are executed in parallel, after which a union of the two result sets +are returned. Sparse vector based hybrid search is not currently supported. + +To execute a hybrid search, your database schema needs to have a vector field and a string field with full text search capabilities enabled. +If you are creating a collection using the Semantic Kernel vector storage connectors, make sure to enable the `IsFullTextSearchable` option +on the string field that you want to target for the keyword search. + +> [!TIP] +> For more information on how to enable `IsFullTextSearchable` refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings) + +## Hybrid Search + +The `HybridSearchAsync` method allows searching using a vector and an `ICollection` of string keywords. It also takes an optional `HybridSearchOptions` class as input. +This method is available on the following interface: + +1. `IKeywordHybridSearch` + +Only connectors for databases that currently support vector plus keyword hybrid search are implementing this interface. + +Assuming you have a collection that already contains data, you can easily do a hybrid search on it. Here is an example using Qdrant. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +IVectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost")); +IKeywordHybridSearch collection = (IKeywordHybridSearch)vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit resulst to the single top match. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], new() { Top = 1 }); + +// Inspect the returned hotel. +await foreach (var record in searchResult.Results) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +``` + +> [!TIP] +> For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). + +## Supported Vector Types + +`HybridSearchAsync` takes a generic type as the vector parameter. +The types of vectors supported by each data store vary. +See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. + +It is also important for the search vector type to match the target vector that is being searched, e.g. if you have two vectors +on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector +you are targeting. +See [VectorProperty and AdditionalProperty](#vectorproperty-and-additionalproperty) for how to pick a target vector if you have more than one per record. + +## Hybrid Search Options + +The following options can be provided using the `HybridSearchOptions` class. + +### VectorProperty and AdditionalProperty + +The `VectorProperty` and `AdditionalProperty` options can be used to specify the vector property and full text search property to target during the search. + +If no `VectorProperty` is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. + +If no `AdditionalProperty` is provided and the data model contains only one full text search property, that property will be used. +If the data model contains no full text search property or multiple full text search properties and `AdditionalProperty` is not provided, the search method will throw. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost")); +var collection = (IKeywordHybridSearch)vectorStore.GetCollection("skproducts"); + +// Create the hybrid search options and indicate that we want +// to search the DescriptionEmbedding vector property and the +// Description full text search property. +var hybridSearchOptions = new HybridSearchOptions +{ + VectorProperty = r => r.DescriptionEmbedding, + AdditionalProperty = r => r.Description +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +public sealed class Product +{ + [VectorStoreRecordKey] + public int Key { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Name { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Description { get; set; } + + [VectorStoreRecordData] + public List FeatureList { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +``` + +### Top and Skip + +The `Top` and `Skip` options allow you to limit the number of results to the Top n results and +to skip a number of results from the top of the resultset. +Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. + +```csharp +// Create the vector search options and indicate that we want to skip the first 40 results and then get the next 20. +var hybridSearchOptions = new HybridSearchOptions +{ + Top = 20, + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.Description); +} +``` + +The default values for `Top` is 3 and `Skip` is 0. + +### IncludeVectors + +The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. +If `false`, the vector properties on the returned model will be left null. +Using `false` can significantly reduce the amount of data retrieved from the vector store during search, +making searches more efficient. + +The default value for `IncludeVectors` is `false`. + +```csharp +// Create the hybrid search options and indicate that we want to include vectors in the search results. +var hybridSearchOptions = new HybridSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.FeatureList); +} +``` + +### Filter + +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection +before applying the vector search. + +This has multiple benefits: + +- Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. +- Limit the resultset for e.g. access control purposes, by excluding data that the user shouldn't have access to. + +Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. +Some vector stores will allow filtering using any field, but may optionally allow indexing to improve filtering performance. + +If creating a collection via the Semantic Kernel vector store abstractions and you wish to enable filtering on a field, +set the `IsFilterable` property to true when defining your data model or when creating your record definition. + +> [!TIP] +> For more information on how to set the `IsFilterable` property, refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings). + +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, e.g. equals, +not equals, and, or, etc. + +```csharp +// Create the hybrid search options and set the filter on the options. +var hybridSearchOptions = new HybridSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreRecordKey] + public ulong Key { get; set; } + + // Category is marked as filterable, since we want to filter using this property. + [VectorStoreRecordData(IsFilterable = true)] + public string Category { get; set; } + + // Tags is marked as filterable, since we want to filter using this property. + [VectorStoreRecordData(IsFilterable = true)] + public List Tags { get; set; } + + [VectorStoreRecordData] + public string Term { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Definition { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +``` + +::: zone-end +::: zone pivot="programming-language-python" + +## Coming soon + +More information coming soon. + +::: zone-end +::: zone pivot="programming-language-java" + +## Coming soon + +More information coming soon. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml index b4c46306..4822cb9d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml @@ -6,10 +6,14 @@ href: azure-cosmosdb-mongodb-connector.md - name: Azure CosmosDB NoSQL connector href: azure-cosmosdb-nosql-connector.md +- name: Chroma connector + href: chroma-connector.md - name: Couchbase connector href: couchbase-connector.md - name: Elasticsearch connector href: elasticsearch-connector.md +- name: Faiss connector + href: faiss-connector.md - name: In-memory connector href: inmemory-connector.md - name: JDBC connector @@ -24,6 +28,8 @@ href: qdrant-connector.md - name: Redis connector href: redis-connector.md +- name: SQL Server connector + href: sql-connector.md - name: SQLite connector href: sqlite-connector.md - name: Volatile (in-memory) connector diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md index 305ce851..869374b0 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md @@ -17,27 +17,65 @@ ms.service: semantic-kernel The Azure AI Search Vector Store connector can be used to access and manage data in Azure AI Search. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure AI Search Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types | ReadOnlyMemory\ | -| Supported index types |
  • Hnsw
  • Flat
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • Hamming
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | + +::: zone-end ## Limitations Notable Azure AI Search connector functionality limitations. -| Feature Area | Workaround | -|--------------------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | +| Feature Area | Workaround | +| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------- | +| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | ::: zone pivot="programming-language-csharp" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md index 46fbbf96..6af3c29d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md @@ -13,30 +13,54 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Azure CosmosDB MongoDB Vector Store connector can be used to access and manage data in Azure CosmosDB MongoDB (vCore). The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Hnsw
  • IvfFlat
| -| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagsEqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | + +::: zone-end +::: zone pivot="programming-language-java" +More info coming soon. +::: zone-end ## Limitations This connector is compatible with Azure Cosmos DB MongoDB (vCore) and is *not* designed to be compatible with Azure Cosmos DB MongoDB (RU). +::: zone pivot="programming-language-csharp" + ## Getting started Add the Azure CosmosDB MongoDB Vector Store connector NuGet package to your project. @@ -165,9 +189,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the Azure CosmosDB MongoDB Vector Store dependencies to your environment. Because the Azure CosmosDB MongoDB connector is built on the MongoDB Atlas connector and uses the same client as that one, you need to install with these extras: + +```bash +pip install semantic-kernel[azure, mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +# If the right environment settings are set, namely AZURE_COSMOS_DB_MONGODB_CONNECTION_STRING and optionally AZURE_COSMOS_DB_MONGODB_DATABASE_NAME, this is enough to create the Store: +store = AzureCosmosDBforMongoDBStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +client = AsyncMongoClient(...) +store = AzureCosmosDBforMongoDBStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = AzureCosmosDBforMongoDBCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the Azure CosmosDB for MongoDB connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md index 6e84f353..4b9807ce 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md @@ -19,19 +19,20 @@ ms.service: semantic-kernel The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure Cosmos DB NoSQL Container | -| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB NoSQL Container | +| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | ## Limitations @@ -247,9 +248,158 @@ var record = await collection.GetAsync(new AzureCosmosDBNoSQLCompositeKey("hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Overview -More info coming soon. +The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB NoSQL Container | +| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| is_filterable supported? | Yes | +| is_full_text_searchable supported? | Yes | +| HybridSearch supported? | No | + +## Getting started + +Add the Azure extra package to your project. + +```bash +pip install semantic-kernel[azure] +``` + +Next you can create a Azure CosmosDB NoSQL Vector Store instance directly. This reads certain environment variables to configure the connection to Azure CosmosDB NoSQL: + +- AZURE_COSMOS_DB_NO_SQL_URL +- AZURE_COSMOS_DB_NO_SQL_DATABASE_NAME + +And optionally: + +- AZURE_COSMOS_DB_NO_SQL_KEY + +When this is not set, a `AsyncDefaultAzureCredential` is used to authenticate. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore() +``` + +You can also supply these values in the constructor: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore( + url="https://.documents.azure.com:443/", + key="", + database_name="" +) +``` + +And you can pass in a CosmosClient instance, just make sure it is a async client. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore +from azure.cosmos.aio import CosmosClient + +client = CosmosClient( + url="https://.documents.azure.com:443/", + credential="" or AsyncDefaultAzureCredential() +) +vector_store = AzureCosmosDBNoSQLStore( + client=client, + database_name="" +) +``` + +The next step needs a data model, a variable called Hotels is used in the example below. + +With a store, you can get a collection: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore() +collection = vector_store.get_collection(collection_name="skhotels", data_model=Hotel) +``` + +It is possible to construct a direct reference to a named collection, this uses the same environment variables as above. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection + +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, +) +``` + +## Using partition key + +In the Azure Cosmos DB for NoSQL connector, the partition key property defaults to the key property - `id`. You can also supply a value for the partition key in the constructor. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection + +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, + partition_key="hotel_name" +) +``` + +This can be a more complex key, when using the `PartitionKey` object: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection +from azure.cosmos import PartitionKey + +partition_key = PartitionKey(path="/hotel_name") +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, + partition_key=partition_key +) +``` + +The `AzureCosmosDBNoSQLVectorStoreRecordCollection` class supports two key types: `string` and `AzureCosmosDBNoSQLCompositeKey`. The `AzureCosmosDBNoSQLCompositeKey` consists of `key` and `partition_key`. + +If the partition key property is not set (and the default key property is used), `string` keys can be used for operations with database records. However, if a partition key property is specified, it is recommended to use `AzureCosmosDBNoSQLCompositeKey` to provide both the key and partition key values to the `get` and `delete` methods. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCompositeKey +from semantic_kernel.data import VectorStoreRecordDataField + +@vectorstoremodel +class data_model_type: + id: Annotated[str, VectorStoreRecordKeyField] + product_type: Annotated[str, VectorStoreRecordDataField()] + ... + +collection = store.get_collection( + collection_name=collection_name, + data_model=data_model_type, + partition_key=PartitionKey(path="/product_type"), +) + +# when there is data in the collection +composite_key = AzureCosmosDBNoSQLCompositeKey( + key='key value', partition_key='partition key value' +) +# get a record, with the partition key +record = await collection.get(composite_key) + +# or delete +await collection.delete(composite_key) +``` ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md new file mode 100644 index 00000000..9cfc7ee8 --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md @@ -0,0 +1,104 @@ +--- +title: Using the Semantic Kernel Chroma Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in ChromaDB. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Using the Chroma connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Not supported + +Not supported. + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The Chroma Vector Store connector can be used to access and manage data in Chroma. The connector has the +following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------ | +| Collection maps to | Chroma collection | +| Supported key property types | string | +| Supported data property types | All types | +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • HNSW
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +## Limitations + +Notable Chroma connector functionality limitations. + +| Feature Area | Workaround | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------- | +| Client-server mode | Use the client.HttpClient and pass the result to the `client` parameter, we do not support a AsyncHttpClient at this time | +| Chroma Cloud | Unclear at this time, as Chroma Cloud is still in private preview | + +## Getting Started + +Add the Chroma Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[chroma] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaStore + +store = ChromaStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from chromadb import Client +from semantic_kernel.connectors.memory.chroma import ChromaStore + +client = Client(...) +store = ChromaStore(client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = ChromaCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Chroma client returns both `get` and `search` results in tabular form, this means that there are between 3 and 5 lists being returned in a dict, the lists are 'keys', 'documents', 'embeddings', and optionally 'metadatas' and 'distances'. The Semantic Kernel Chroma connector will automatically convert this into a list of `dict` objects, which are then parsed back to your data model. + +It could be very interesting performance wise to do straight serialization from this format into a dataframe-like structure as that saves a lot of rebuilding of the data structure. This is not done for you, even when using container mode, you would have to specify this yourself, for more details on this concept see the [serialization documentation](./../serialization.md). + +::: zone-end +::: zone pivot="programming-language-java" + +## Not supported + +Not supported. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md index 63ba111f..e27ad81a 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md @@ -34,6 +34,7 @@ following characteristics. | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | ## Getting Started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md index 632ea9f8..6f41b9e3 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md @@ -13,6 +13,8 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. +::: zone pivot="programming-language-csharp" + ## Overview The Elasticsearch Vector Store connector can be used to access and manage data in Elasticsearch. The connector has the following characteristics. @@ -30,8 +32,7 @@ The Elasticsearch Vector Store connector can be used to access and manage data i | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | - -::: zone pivot="programming-language-csharp" +| HybridSearch supported? | No | ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md new file mode 100644 index 00000000..7fc2c4a2 --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md @@ -0,0 +1,121 @@ +--- +title: Using the Semantic Kernel Faiss Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in an in-memory Faiss vector store. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 03/13/2025 +ms.service: semantic-kernel +--- +# Using the Faiss connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Not supported at this time + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The [Faiss](https://github.com/facebookresearch/faiss) Vector Store connector is a Vector Store implementation provided by Semantic Kernel that uses no external database and stores data in memory and vectors in a Faiss Index. It uses the [`InMemoryVectorCollection`](./inmemory-connector.md) for the other parts of the records, while using the Faiss indexes for search. +This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. + +The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | In-memory and Faiss indexes dictionary | +| Supported key property types | Any that is allowed to be a dict key, see the python documentation for details [here](https://docs.python.org/3/library/stdtypes.html#typesmapping) | +| Supported data property types | Any type | +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types | Flat (see [custom indexes](#custom-indexes)) | +| Supported distance functions |
  • Dot Product Similarity
  • Euclidean Squared Distance
| +| Supports multiple vectors in a record | Yes | +| is_filterable supported? | Yes | +| is_full_text_searchable supported? | Yes | + +## Getting started + +Add the Semantic Kernel package to your project. + +```cmd +pip install semantic-kernel[faiss] +``` + +In the snippets below, it is assumed that you have a data model class defined named 'DataModel'. + +```python +from semantic_kernel.connectors.memory.faiss import FaissStore + +vector_store = FaissStore() +vector_collection = vector_store.get_collection("collection_name", DataModel) +``` + +It is possible to construct a direct reference to a named collection. + +```python +from semantic_kernel.connectors.memory.faiss import FaissCollection + +vector_collection = FaissCollection("collection_name", DataModel) +``` + +## Custom indexes + +The Faiss connector is limited to the Flat index type. + +Given the complexity of Faiss indexes, you are free to create your own index(es), including building the faiss-gpu package and using indexes from that. When doing this, any metrics defined on a vector field is ignored. If you have multiple vectors in your datamodel, you can pass in custom indexes only for the ones you want and let the built-in indexes be created, with a flat index and the metric defined in the model. + +Important to note, if the index requires training, then make sure to do that as well, whenever we use the index, a check is done on the `is_trained` attribute of the index. + +The index is always available (custom or built-in) in the `indexes` property of the collection. You can use this to get the index and do any operations you want on it, so you can do training afterwards, just make sure to do that before you want to use any CRUD against it. + +To pass in your custom index, use either: + +```python + +import faiss + +from semantic_kernel.connectors.memory.faiss import FaissCollection + +index = faiss.IndexHNSW(d=768, M=16, efConstruction=200) # or some other index +vector_collection = FaissCollection( + collection_name="collection_name", + data_model_type=DataModel, + indexes={"vector_field_name": index} +) +``` + +or: + +```python + +import faiss + +from semantic_kernel.connectors.memory.faiss import FaissCollection + +index = faiss.IndexHNSW(d=768, M=16, efConstruction=200) # or some other index +vector_collection = FaissCollection( + collection_name="collection_name", + data_model_type=DataModel, +) +await vector_collection.create_collection( + indexes={"vector_field_name": index} +) +# or when you have only one vector field: +await vector_collection.create_collection( + index=index +) + +``` + +::: zone-end +::: zone pivot="programming-language-java" + +## Not supported at this time + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md index 15b30d1d..11721759 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md @@ -22,60 +22,61 @@ Semantic Kernel provides a number of out-of-the-box Vector Store integrations ma ::: zone pivot="programming-language-csharp" -| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:---------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | -| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :--------------------------: | :----------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | +| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | +| Chroma | Planned | | | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Milvus | Planned | | | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Neon |Use [Postgres](./postgres-connector.md)| ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Sql Server | Planned | | | +| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-python" -| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :-----: | :----------------------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Chroma](./chroma-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | +| [Faiss](./faiss-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Neon |Use [Postgres](./postgres-connector.md)| ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [SQL Server](./sql-connector.md) | ✅ | [pyodbc](https://pypi.org/project/pyodbc/) | Microsoft Semantic Kernel Project | +| SQLite | Planned | | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-java" -| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:--------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------- | :-----------------------------: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | ::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md index 1a08b629..fd626301 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md @@ -35,6 +35,7 @@ The connector has the following characteristics. | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | No, since storage is in-memory and data reuse is therefore not possible, custom naming is not applicable. | +| HybridSearch supported? | No | ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md index 6c406610..7591b678 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md @@ -13,25 +13,50 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The MongoDB Vector Store connector can be used to access and manage data in MongoDB. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | MongoDB Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types | N/A | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types | N/A | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagsEqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | + +::: zone-end +::: zone pivot="programming-language-java" + +More info coming soon. + +::: zone-end +::: zone pivot="programming-language-csharp" ## Getting started @@ -136,9 +161,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the MongoDB Atlas Vector Store dependencies to your environment. It needs the `pymongo` package which is included in the mongo extra: , you need to install with these extras: + +```bash +pip install semantic-kernel[mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +# If the right environment settings are set, namely MONGODB_ATLAS_CONNECTION_STRING and optionally MONGODB_ATLAS_DATABASE_NAME and MONGODB_ATLAS_INDEX_NAME, this is enough to create the Store: +store = MongoDBAtlasStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +client = AsyncMongoClient(...) +store = MongoDBAtlasStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = MongoDBAtlasCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the MongoDB Atlas connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md index 134a29bc..cbd94d3b 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md @@ -19,19 +19,21 @@ ms.service: semantic-kernel The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Pinecone serverless Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| -| Supported vector property types | ReadOnlyMemory\ | -| Supported index types | PGA (Pinecone Graph Algorithm) | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | No | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | Yes | +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| Collection maps to | Pinecone serverless Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types | PGA (Pinecone Graph Algorithm) | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | +| Integrated Embeddings supported? | No | ## Getting started @@ -174,7 +176,133 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -The Pinecone connector is not yet available in Python. +## Overview + +The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Pinecone serverless Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types | PGA (Pinecone Graph Algorithm) | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagEqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| Integrated Embeddings supported? | Yes, see [here](#integrated-embeddings) | +| GRPC Supported? | Yes, see [here](#grpc-support) | + +## Getting started + +Add the Pinecone Vector Store connector extra to your project. + +```bash +pip install semantic-kernel[pinecone] +``` + +You can then create a PineconeStore instance and use it to create a collection. +This will read the Pinecone API key from the environment variable `PINECONE_API_KEY`. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore + +store = PineconeStore() +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +It is possible to construct a direct reference to a named collection. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +``` + +You can also create your own Pinecone client and pass it into the constructor. +The client needs to be either `PineconeAsyncio` or `PineconeGRPC` (see [GRPC Support](#grpc-support)). + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore, PineconeCollection +from pinecone import PineconeAsyncio + +client = PineconeAsyncio(api_key="your_api_key") +store = PineconeStore(client=client) +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +### GRPC support + +We also support two options on the collection constructor, the first is to enable GRPC support: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel, use_grpc=True) +``` + +Or with your own client: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore +from pinecone.grpc import PineconeGRPC + +client = PineconeGRPC(api_key="your_api_key") +store = PineconeStore(client=client) +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +### Integrated Embeddings + +The second is to use the integrated embeddings of Pinecone, this will check for a environment variable called `PINECONE_EMBED_MODEL` with the model name, or you can pass in a `embed_settings` dict, which can contain just the model key, or the full settings for the embedding model. In the former case, the other settings will be derived from the data model definition. + +See [Pinecone docs](https://docs.pinecone.io/guides/indexes/create-an-index) and then the `Use integrated embeddings` sections. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +``` + +Alternatively, when not settings the environment variable, you can pass the embed settings into the constructor: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel, embed_settings={"model": "multilingual-e5-large"}) +``` + +This can include other details about the vector setup, like metric and field mapping. +You can also pass the embed settings into the `create_collection` method, this will override the default settings set during initialization. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +await collection.create_collection(embed_settings={"model": "multilingual-e5-large"}) +``` + +> Important: GRPC and Integrated embeddings cannot be used together. + +## Index Namespace + +The Vector Store abstraction does not support a multi tiered record grouping mechanism. Collections in the abstraction map to a Pinecone serverless index +and no second level exists in the abstraction. Pinecone does support a second level of grouping called namespaces. + +By default the Pinecone connector will pass `''` as the namespace for all operations. However it is possible to pass a single namespace to the +Pinecone collection when constructing it and use this instead for all operations. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection( + collection_name="collection_name", + data_model=DataModel, + namespace="seasidehotels" +) +``` ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 8eb24e45..612ce262 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -17,7 +17,9 @@ ms.service: semantic-kernel ## Overview -The Postgres Vector Store connector can be used to access and manage data in Postgres. The connector has the following characteristics. +The Postgres Vector Store connector can be used to access and manage data in Postgres and also supports [Neon Serverless Postgres](https://neon.tech/). + +The connector has the following characteristics. | Feature Area | Support | |-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| @@ -32,6 +34,22 @@ The Postgres Vector Store connector can be used to access and manage data in Pos | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | + +## Limitations + +> [!IMPORTANT] +> When initializing `NpgsqlDataSource` manually, it is necessary to call `UseVector` on the `NpgsqlDataSourceBuilder`. This enables vector support. Without this, usage of the VectorStore implementation will fail. + +Here is an example of how to call `UseVector`. + +```csharp +NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +dataSourceBuilder.UseVector(); +NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +``` + +When using the `AddPostgresVectorStore` dependency injection registration method with a connection string, the datasource will be constructed by this method and will automatically have `UseVector` applied. ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md index 03ee4e4b..bd6adebc 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md @@ -17,6 +17,8 @@ ms.service: semantic-kernel The Qdrant Vector Store connector can be used to access and manage data in Qdrant. The connector has the following characteristics. +::: zone pivot="programming-language-csharp" + | Feature Area | Support | |-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | Collection maps to | Qdrant collection with payload indices for filterable data fields | @@ -30,6 +32,33 @@ The Qdrant Vector Store connector can be used to access and manage data in Qdran | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| +| Collection maps to | Qdrant collection with payload indices for filterable data fields | +| Supported key property types |
  • ulong
  • Guid
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
| +| Supported index types | Hnsw | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes (configurable) | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" + +## Not Supported + +Not currently supported. + +::: zone-end ::: zone pivot="programming-language-csharp" @@ -203,6 +232,8 @@ For more details on this concept see the [serialization documentation](./../seri ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" + ### Qdrant vector modes Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. @@ -213,8 +244,6 @@ The default mode is *single unnamed vector*. With this option a collection may only contain a single vector and it will be unnamed in the storage model in Qdrant. Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: -::: zone pivot="programming-language-csharp" - ```csharp new Hotel { @@ -236,6 +265,16 @@ new Hotel ::: zone-end ::: zone pivot="programming-language-python" +### Qdrant vector modes + +Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. +The default mode is *single unnamed vector*. + +#### Single unnamed vector + +With this option a collection may only contain a single vector and it will be unnamed in the storage model in Qdrant. +Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: + ```python Hotel( hotel_id = 1, @@ -254,18 +293,18 @@ PointStruct( vector=[0.9, 0.1, 0.1, 0.1], ) ``` + ::: zone-end ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" #### Named vectors If using the named vectors mode, it means that each point in a collection may contain more than one vector, and each will be named. Here is an example of how an object is represented in Qdrant when using *named vectors* mode: -::: zone pivot="programming-language-csharp" - ```csharp new Hotel { @@ -291,6 +330,11 @@ new Hotel ::: zone-end ::: zone pivot="programming-language-python" +#### Named vectors + +If using the named vectors mode, it means that each point in a collection may contain more than one vector, and each will be named. +Here is an example of how an object is represented in Qdrant when using *named vectors* mode: + ```python Hotel( hotel_id = 1, @@ -318,11 +362,11 @@ PointStruct( ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" + To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. The same options can also be passed to any of the provided dependency injection container extension methods. -::: zone pivot="programming-language-csharp" - ```csharp using Microsoft.SemanticKernel.Connectors.Qdrant; using Qdrant.Client; @@ -340,6 +384,9 @@ var collection = new QdrantVectorStoreRecordCollection( ::: zone-end ::: zone pivot="programming-language-python" +To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. +The same options can also be passed to any of the provided dependency injection container extension methods. + In python the default value for `named_vectors` is True, but you can also disable this as shown below. ```python @@ -351,6 +398,7 @@ collection = QdrantCollection( named_vectors=False, ) ``` + ::: zone-end ::: zone pivot="programming-language-java" ::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md index 72d8ef82..f1436bc9 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md @@ -32,6 +32,7 @@ The connector has the following characteristics. | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | **When using Hashes:** Yes
**When using JSON:** No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | ::: zone pivot="programming-language-csharp" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sql-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sql-connector.md new file mode 100644 index 00000000..ef59a1c9 --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sql-connector.md @@ -0,0 +1,121 @@ +--- +title: Using the Semantic Kernel SQL Server Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in SQL Server. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 03/21/2024 +ms.service: semantic-kernel +--- +# Using the SQL Server Vector Store connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Coming soon + +More info coming soon. + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The [SQL Server](/sql) Vector Store connector is a Vector Store implementation provided by Semantic Kernel that uses Azure SQL as a vector store. Once SQL Server on-prem supports vectors it can also be used with that. + +The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------- | +| Collection maps to | Table dictionary | +| Supported key property types |
  • str
  • int
| +| Supported data property types | Any type | +| Supported vector property types |
  • list[float]
  • numpy array
| +| Supported index types |
  • Flat
| +| Supported distance functions |
  • Cosine Distance
  • Dot Product Similarity
  • Euclidean Distance
| +| Supports multiple vectors in a record | Yes | +| is_filterable supported? | Yes | +| is_full_text_searchable supported? | No | + +## Getting started + +Add the Semantic Kernel package to your project. + +```bash +pip install semantic-kernel[sql] +``` + +The SQL Server connector uses the [pyodbc](https://pypi.org/project/pyodbc/) package to connect to SQL Server. The extra will install the package, but you will need to install the ODBC driver for SQL Server separately, this differs by platform, see the [Azure SQL Documentation](/azure/azure-sql/database/azure-sql-python-quickstart) for details. + +In order for the store and collection to work, it needs a connection string, this can be passed to the constructor or be set in the environment variable `SQL_SERVER_CONNECTION_STRING`. In order to properly deal with vectors, the `LongAsMax=yes` option will be added if not found. It also can use both username/password or integrated security, for the latter, the `DefaultAzureCredential` is used. + +In the snippets below, it is assumed that you have a data model class defined named 'DataModel'. + +```python +from semantic_kernel.connectors.memory.sql_server import SqlServerStore + +vector_store = SqlServerStore() + +# OR + +vector_store = SqlServerStore(connection_string="Driver={ODBC Driver 18 for SQL Server};Server=server_name;Database=database_name;UID=user;PWD=password;LongAsMax=yes;") + +vector_collection = vector_store.get_collection("dbo.table_name", DataModel) +``` + +It is possible to construct a direct reference to a named collection. + +```python +from semantic_kernel.connectors.memory.sql_server import SqlServerCollection + +vector_collection = SqlServerCollection("dbo.table_name", DataModel) +``` + +> Note: The collection name can be specified as a simple string (e.g. `table_name`) or as a fully qualified name (e.g. `dbo.table_name`). The latter is recommended to avoid ambiguity, if no schema is specified, the default schema (`dbo`) will be used. + +When you have specific requirements for the connection, you can also pass in a `pyodbc.Connection` object to the `SqlServerStore` constructor. This allows you to use a custom connection string or other connection options: + +```python +from semantic_kernel.connectors.memory.sql_server import SqlServerStore +import pyodbc + +# Create a connection to the SQL Server database +connection = pyodbc.connect("Driver={ODBC Driver 18 for SQL Server};Server=server_name;Database=database_name;UID=user;PWD=password;LongAsMax=yes;") +# Create a SqlServerStore with the connection +vector_store = SqlServerStore(connection=connection) +``` + +You will have to make sure to close the connection yourself, as the store or collection will not do that for you. + +## Custom create queries + +The SQL Server connector is limited to the Flat index type. + +The `create_collection` method on the `SqlServerCollection` allows you to pass in a single or multiple custom queries to create the collection. The queries are executed in the order they are passed in, no results are returned. + +If this is done, there is no guarantee that the other methods still work as expected. The connector is not aware of the custom queries and will not validate them. + +If the `DataModel` has `id`, `content`, and `vector` as fields, then for instance you could create the table like this in order to also create a index on the content field: + +```python +from semantic_kernel.connectors.memory.sql_server import SqlServerCollection + +# Create a collection with a custom query +async with SqlServerCollection("dbo.table_name", DataModel) as collection: + collection.create_collection( + queries=["CREATE TABLE dbo.table_name (id INT PRIMARY KEY, content NVARCHAR(3000) NULL, vector VECTOR(1536) NULL ) PRIMARY KEY (id);", + "CREATE INDEX idx_content ON dbo.table_name (content);"] + ) +``` + +::: zone-end +::: zone pivot="programming-language-java" + +## Coming soon + +More info coming soon. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md index 0e1d9c25..5f13fcf8 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md @@ -32,6 +32,7 @@ The SQLite Vector Store connector can be used to access and manage data in SQLit | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | ## Limitations diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md index 52a0178a..7b30706d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md @@ -13,33 +13,55 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Weaviate Collection | -| Supported key property types | Guid | -| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| -| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" +Coming soon. +::: zone-end ## Limitations Notable Weaviate connector functionality limitations. -| Feature Area | Workaround | -|------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | +| Feature Area | Workaround | +| ---------------------------------------------------------------------- | --------------------------------------------------- | +| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | > [!WARNING] > Weaviate requires collection names to start with an upper case letter. If you do not provide a collection name with an upper case letter, Weaviate will return an error when you try and create your collection. The error that you will see is `Cannot query field "mycollection" on type "GetObjectsObj". Did you mean "Mycollection"?` where `mycollection` is your collection name. In this example, if you change your collection name to `Mycollection` instead, this will fix the error. @@ -181,9 +203,64 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting Started -More info coming soon. +Add the Weaviate Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[weaviate] +``` + +You can then create the vector store, it uses environment settings to connect: + +For using Weaviate Cloud: + +- url: WEAVIATE_URL +- api_key: WEAVIATE_API_KEY + +For using Weaviate Local (i.e. Weaviate in a Docker container): + +- local_host: WEAVIATE_LOCAL_HOST +- local_port: WEAVIATE_LOCAL_PORT +- local_grpc_port: WEAVIATE_LOCAL_GRPC_PORT + +If you want to use embedded: + +- use_embed: WEAVIATE_USE_EMBED + +These should be set exclusively, so only one set of the above is present, otherwise it will raise an exception. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +store = WeaviateStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +import weaviate +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +client = weaviate.WeaviateAsyncClient(...) +store = WeaviateStore(async_client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = WeaviateCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Weaviate client returns it's own objects which are parsed and turned into dicts in the regular flow, for more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/vector-search.md b/semantic-kernel/concepts/vector-store-connectors/vector-search.md index 27d71481..8d6855ac 100644 --- a/semantic-kernel/concepts/vector-store-connectors/vector-search.md +++ b/semantic-kernel/concepts/vector-store-connectors/vector-search.md @@ -19,7 +19,7 @@ Semantic Kernel provides vector search capabilities as part of its Vector Store ## Vector Search -The `VectorizedSearchAsync` method allows searching using data that has already been vectorized. This method takes a vector and an optional `VectorSearchOptions` class as input. +The `VectorizedSearchAsync` method allows searching using data that has already been vectorized. This method takes a vector and an optional `VectorSearchOptions` class as input. This method is available on the following interfaces: 1. `IVectorizedSearch` @@ -64,26 +64,23 @@ await foreach (var record in searchResult.Results) ## Supported Vector Types `VectorizedSearchAsync` takes a generic type as the vector parameter. -The types of vectors supported y each data store vary. +The types of vectors supported by each data store vary. See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. It is also important for the search vector type to match the target vector that is being searched, e.g. if you have two vectors on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector you are targeting. -See [VectorPropertyName](#vectorpropertyname) for how to pick a target vector if you have more than one per record. +See [VectorProperty](#vectorproperty) for how to pick a target vector if you have more than one per record. ## Vector Search Options -The following options can be provided using the `VectorSearchOptions` class. - -### VectorPropertyName +The following options can be provided using the `VectorSearchOptions` class. -The `VectorPropertyName` option can be used to specify the name of the vector property to target during the search. -If none is provided, the first vector found on the data model or specified in the record definition will be used. +### VectorProperty -Note that when specifying the `VectorPropertyName`, use the name of the property as defined on the data model or in the record definition. -Use this property name even if the property may be stored under a different name in the vector store. The storage name may e.g. be different -because of custom serialization settings. +The `VectorProperty` option can be used to specify the vector property to target during the search. +If none is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. ```csharp using Microsoft.Extensions.VectorData; @@ -93,13 +90,13 @@ var vectorStore = new InMemoryVectorStore(); var collection = vectorStore.GetCollection("skproducts"); // Create the vector search options and indicate that we want to search the FeatureListEmbedding property. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { - VectorPropertyName = nameof(Product.FeatureListEmbedding) + VectorProperty = r => r.FeatureListEmbedding }; // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = await collection.VectorizedSearchAsync(searchVector, vectorSearchOptions).Results.ToListAsync(); +var searchResult = await collection.VectorizedSearchAsync(searchVector, vectorSearchOptions); public sealed class Product { @@ -128,7 +125,7 @@ Top and Skip can be used to do paging if you wish to retrieve a large number of ```csharp // Create the vector search options and indicate that we want to skip the first 40 results and then get the next 20. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { Top = 20, Skip = 40 @@ -157,7 +154,7 @@ The default value for `IncludeVectors` is `false`. ```csharp // Create the vector search options and indicate that we want to include vectors in the search results. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { IncludeVectors = true }; @@ -172,9 +169,9 @@ await foreach (var result in searchResult.Results) } ``` -### VectorSearchFilter +### Filter -The `VectorSearchFilter` option can be used to provide a filter for filtering the records in the chosen collection +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection before applying the vector search. This has multiple benefits: @@ -191,22 +188,16 @@ set the `IsFilterable` property to true when defining your data model or when cr > [!TIP] > For more information on how to set the `IsFilterable` property, refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings). -To create a filter use the `VectorSearchFilter` class. You can combine multiple filter clauses together in one `VectorSearchFilter`. -All filter clauses are combined with `and`. -Note that when providing a property name when constructing the filter, use the name of the property as defined on the data model or in the record definition. -Use this property name even if the property may be stored under a different name in the vector store. The storage name may e.g. be different -because of custom serialization settings. +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, e.g. equals, +not equals, and, or, etc. ```csharp -// Filter where Category == 'External Definitions' and Tags contain 'memory'. -var filter = new VectorSearchFilter() - .EqualTo(nameof(Glossary.Category), "External Definitions") - .AnyTagEqualTo(nameof(Glossary.Tags), "memory"); - // Create the vector search options and set the filter on the options. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { - Filter = filter + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") }; // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. @@ -242,15 +233,6 @@ sealed class Glossary } ``` -#### EqualTo filter clause - -Use `EqualTo` for a direct comparison between property and value. - -#### AnyTagEqualTo filter clause - -Use `AnyTagEqualTo` to check if any of the strings, stored in a tag property in the vector store, contains a provided value. -For a property to be considered a tag property, it needs to be a List, array or other enumerable of string. - ::: zone-end ::: zone pivot="programming-language-python" diff --git a/semantic-kernel/index.yml b/semantic-kernel/index.yml index 3f94c798..b8a4c02f 100644 --- a/semantic-kernel/index.yml +++ b/semantic-kernel/index.yml @@ -51,7 +51,7 @@ productDirectory: links: - url: /semantic-kernel/concepts/enterprise-readiness/observability/index text: Observability - - url: /semantic-kernel/concepts/enterprise-readiness/SECURITY + - url: /semantic-kernel/support/SECURITY.md text: Security - url: /semantic-kernel/concepts/enterprise-readiness/filters text: Filters diff --git a/semantic-kernel/media/agentSKdocs.png b/semantic-kernel/media/agentSKdocs.png new file mode 100644 index 00000000..e90d9fa4 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs.png differ diff --git a/semantic-kernel/media/agentSKdocs2.png b/semantic-kernel/media/agentSKdocs2.png new file mode 100644 index 00000000..1d897fc6 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs2.png differ diff --git a/semantic-kernel/media/agentSKdocs3.png b/semantic-kernel/media/agentSKdocs3.png new file mode 100644 index 00000000..e7fee71f Binary files /dev/null and b/semantic-kernel/media/agentSKdocs3.png differ diff --git a/semantic-kernel/media/agentSKdocs4.png b/semantic-kernel/media/agentSKdocs4.png new file mode 100644 index 00000000..816ada71 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs4.png differ diff --git a/semantic-kernel/support/migration/agent-framework-rc-migration-guide.md b/semantic-kernel/support/migration/agent-framework-rc-migration-guide.md index a7eb9cfd..88c7ca7e 100644 --- a/semantic-kernel/support/migration/agent-framework-rc-migration-guide.md +++ b/semantic-kernel/support/migration/agent-framework-rc-migration-guide.md @@ -15,7 +15,135 @@ As we transition some agents from the experimental stage to the release candidat ::: zone pivot="programming-language-csharp" -## OpenAIAssistantAgent C# Migration Guide +## Common Agent Invocation API + +In version 1.43.0 we are releasing a new common agent invocation API, that will allow all agent types to be invoked via a common API. + +To enable this new API we are introducing the concept of an `AgentThread`, which represents a conversation thread and abstracts away the different thread management requirements of different agent types. For some agent types it will also, in future, allow different thread imlementations to be used with the same agent. + +The common `Invoke` methods that we are introducing allow you to provide the message(s) that you want to pass to the agent and an optional `AgentThread`. If an `AgentThread` is provided, this will continue the conversation already on the `AgentThread`. If no `AgentThread` is provided, a new default thread will be created and returned as part of the response. + +It is also possible to manually create an `AgentThread` instance, for example in cases where you may have a thread id from the underlying agent service, and you want to continue that thread. You may also want to customize the options for the thread, e.g. associate tools. + +Here is a simple example of how any agent can now be used with agent agnostic code. + +```csharp +private async Task UseAgentAsync(Agent agent, AgentThread? agentThread = null) +{ + // Invoke the agent, and continue the existing thread if provided. + var responses = agent.InvokeAsync(new ChatMessageContent(AuthorRole.User, "Hi"), agentThread); + + // Output results. + await foreach (AgentResponseItem response in responses) + { + Console.WriteLine(response); + agentThread = response.Thread; + } + + // Delete the thread if required. + if (agentThread is not null) + { + await agentThread.DeleteAsync(); + } +} +``` + +These changes were applied in: + +- [PR #11116](https://github.com/microsoft/semantic-kernel/pull/11116) + +### Azure AI Agent Thread Options + +The `AzureAIAgent` currently only supports threads of type `AzureAIAgentThread`. + +In addition to allowing a thread to be created for you automatically on agent invocation, you can also manually +construct an instance of an `AzureAIAgentThread`. + +`AzureAIAgentThread` supports being created with customized tools and metadata, plus messages to seed the conversation with. + +```csharp +AgentThread thread = new AzureAIAgentThread( + agentsClient, + messages: seedMessages, + toolResources: tools, + metadata: metadata); +``` + +You can also construct an instance of an `AzureAIAgentThread` that continues an existing conversation. + +```csharp +AgentThread thread = new AzureAIAgentThread( + agentsClient, + id: "my-existing-thread-id"); +``` + +### Bedrock Agent Thread Options + +The `BedrockAgent` currently only supports threads of type `BedrockAgentThread`. + +In addition to allowing a thread to be created for you automatically on agent invocation, you can also manually +construct an instance of an `BedrockAgentThread`. + +```csharp +AgentThread thread = new BedrockAgentThread(amazonBedrockAgentRuntimeClient); +``` + +You can also construct an instance of an `BedrockAgentThread` that continues an existing conversation. + +```csharp +AgentThread thread = new BedrockAgentThread( + amazonBedrockAgentRuntimeClient, + sessionId: "my-existing-session-id"); +``` + +### Chat Completion Agent Thread Options + +The `ChatCompletionAgent` currently only supports threads of type `ChatHistoryAgentThread`. +`ChatHistoryAgentThread` uses an in-memory `ChatHistory` object to store the messages on the thread. + +In addition to allowing a thread to be created for you automatically on agent invocation, you can also manually +construct an instance of an `ChatHistoryAgentThread`. + +```csharp +AgentThread thread = new ChatHistoryAgentThread(); +``` + +You can also construct an instance of an `ChatHistoryAgentThread` that continues an existing conversation +by passing in a `ChatHistory` object with the existing messages. + +```csharp +ChatHistory chatHistory = new([new ChatMessageContent(AuthorRole.User, "Hi")]); + +AgentThread thread = new ChatHistoryAgentThread(chatHistory: chatHistory); +``` + +### OpenAI Assistant Thread Options + +The `OpenAIAssistantAgent` currently only supports threads of type `OpenAIAssistantAgentThread`. + +In addition to allowing a thread to be created for you automatically on agent invocation, you can also manually +construct an instance of an `OpenAIAssistantAgentThread`. + +`OpenAIAssistantAgentThread` supports being created with customized tools and metadata, plus messages to seed the conversation with. + +```csharp +AgentThread thread = new OpenAIAssistantAgentThread( + assistantClient, + messages: seedMessages, + codeInterpreterFileIds: fileIds, + vectorStoreId: "my-vector-store", + metadata: metadata); +``` + +You can also construct an instance of an `OpenAIAssistantAgentThread` that continues an existing conversation. + +```csharp +AgentThread thread = new OpenAIAssistantAgentThread( + assistantClient, + id: "my-existing-thread-id"); +``` + +## OpenAIAssistantAgent C# Migration Guide We recently applied a significant shift around the [`OpenAIAssistantAgent`](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/OpenAI/OpenAIAssistantAgent.cs) in the _Semantic Kernel Agent Framework_. @@ -104,24 +232,29 @@ await assistantClient.DeleteAssistantAsync(agent.Id); ## 5. Thread Lifecycle ### **Creating a Thread** -Threads are now created directly using `AssistantClient`. + +Threads are now managed via `AssistantAgentThread`. ##### **New Way** -```csharp -AssistantThread thread = await assistantClient.CreateThreadAsync(); -``` -Using a convenience extension method: ```csharp -string threadId = await assistantClient.CreateThreadAsync(messages: [new ChatMessageContent(AuthorRole.User, "")]); +var thread = new AssistantAgentThread(assistantClient); +// Calling CreateAsync is an optional step. +// A thread will be created automatically on first use if CreateAsync was not called. +// Note that CreateAsync is not on the AgentThread base implementation since not all +// agent services support explicit thread creation. +await thread.CreateAsync(); ``` ##### **Old Way (Deprecated)** + Previously, thread management was indirect or agent-bound. ### **Thread Deletion** + ```csharp -await assistantClient.DeleteThreadAsync(thread.Id); +var thread = new AssistantAgentThread(assistantClient, "existing-thread-id"); +await thread.DeleteAsync(); ``` ## 6. File Lifecycle @@ -165,12 +298,18 @@ Deprecated patterns are marked with `[Obsolete]`. To suppress obsolete warnings This migration guide helps you transition smoothly to the new implementation, simplifying client initialization, resource management, and integration with the **Semantic Kernel .NET SDK**. ::: zone-end + ::: zone pivot="programming-language-python" -For developers upgrading to Semantic Kernel Python 1.22.0 or later, the ChatCompletionAgent and OpenAI Assistant abstractions have been updated. +> [!IMPORTANT] +> For developers upgrading to Semantic Kernel Python 1.26.1 or later, significant updates and breaking changes have been introduced to improve our agent framework as we approach GA. These changes were applied in: +- [PR #11116](https://github.com/microsoft/semantic-kernel/pull/11116) + +Previous changes were applied in: + - [PR #10666](https://github.com/microsoft/semantic-kernel/pull/10666) - [PR #10667](https://github.com/microsoft/semantic-kernel/pull/10667) - [PR #10701](https://github.com/microsoft/semantic-kernel/pull/10701) @@ -178,6 +317,368 @@ These changes were applied in: This guide provides step-by-step instructions for migrating your Python code from the old implementation to the new implementation. +## Agent Imports + +All agent import paths have been consolidated under `semantic_kernel.agents`. + +#### Updated import style + +```python +from semantic_kernel.agents import ( + AutoGenConversableAgent, + AzureAIAgent, + AzureAssistantAgent, + BedrockAgent, + ChatCompletionAgent, + OpenAIAssistantAgent, +) +``` + +#### Previous import style (deprecated): + +``` +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.agents.autogen import AutoGenConversableAgent +from semantic_kernel.agents.azure_ai import AzureAIAgent +from semantic_kernel.agents.bedrock import BedrockAgent +from semantic_kernel.agents.open_ai import AzureAssistantAgent, OpenAIAssistantAgent +``` + +## Common Agent Invocation API + +As of Semantic Kernel Python 1.26.0 and later, we introduced a new common abstraction to manage threads for all agents. For each agent we now expose a thread class that implements the `AgentThread` base class, allowing context management via methods like `create()` and `delete()`. + +Agent responses `get_response(...)`, `invoke(...)`, `invoke_stream(...)` now return an `AgentResponseItem[ChatMessageContent]`, which has two attributes: + +```python +message: TMessage # Usually ChatMessageContent +thread: AgentThread # Contains the concrete type for the given agent +``` + +### Adding Messages to a Thread + +Messages should be added to a thread via the `messages` argument as part of the agent's `get_response(...)`, `invoke(...)` or `invoke_stream(...)` methods. + +### Azure AI Agent Thread + +An `AzureAIAgentThread` can be created as follows: + +```python +from semantic_kernel.agents import AzureAIAgentThread + +thread = AzureAIAgentThread( + client: AIProjectClient, # required + messages: list[ThreadMessageOptions] | None = None, # optional + metadata: dict[str, str] | None = None, # optional + thread_id: str | None = None, # optional + tool_resources: "ToolResources | None" = None, # optional +) +``` + +Providing a `thread_id` (string) allows you to continue an existing conversation. If omitted, a new thread is created and returned as part of the agent response. + +A complete implementation example: + +```python +import asyncio + +from azure.identity.aio import DefaultAzureCredential + +from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread + +USER_INPUTS = [ + "Why is the sky blue?", + "What are we talking about?", +] + +async def main() -> None: + ai_agent_settings = AzureAIAgentSettings.create() + + async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + # 1. Create an agent on the Azure AI agent service + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + name="Assistant", + instructions="Answer the user's questions.", + ) + + # 2. Create a Semantic Kernel agent for the Azure AI agent + agent = AzureAIAgent( + client=client, + definition=agent_definition, + ) + + # 3. Create a thread for the agent + # If no thread is provided, a new thread will be + # created and returned with the initial response + thread: AzureAIAgentThread = None + + try: + for user_input in USER_INPUTS: + print(f"# User: {user_input}") + # 4. Invoke the agent with the specified message for response + response = await agent.get_response(messages=user_input, thread=thread) + print(f"# {response.content}: {response}") + thread = response.thread + finally: + # 6. Cleanup: Delete the thread and agent + await thread.delete() if thread else None + await client.agents.delete_agent(agent.id) + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### Bedrock Agent Thread + +A `BedrockAgent` uses a `BedrockAgentThread` to manage conversation history and context. You may provide a `session_id` to either continue or initiate a fresh conversation context. + +```python +from semantic_kernel.agents import BedrockAgentThread + +thread = BedrockAgentThread( + bedrock_runtime_client: Any, + session_id: str | None = None, +) +``` + +If no `session_id` is provided, a new context is created automatically. + +A complete implementation example: + +```python +import asyncio + +from semantic_kernel.agents import BedrockAgent, BedrockAgentThread + +async def main(): + bedrock_agent = await BedrockAgent.create_and_prepare_agent( + "semantic-kernel-bedrock-agent", + instructions="You are a friendly assistant. You help people find information.", + ) + + # Create a thread for the agent + # If no thread is provided, a new thread will be + # created and returned with the initial response + thread: BedrockAgentThread = None + + try: + while True: + user_input = input("User:> ") + if user_input == "exit": + print("\n\nExiting chat...") + break + + # Invoke the agent + # The chat history is maintained in the session + response = await bedrock_agent.get_response( + input_text=user_input, + thread=thread, + ) + print(f"Bedrock agent: {response}") + thread = response.thread + except KeyboardInterrupt: + print("\n\nExiting chat...") + return False + except EOFError: + print("\n\nExiting chat...") + return False + finally: + # Delete the agent + await bedrock_agent.delete_agent() + await thread.delete() if thread else None + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### Chat History Agent Thread + +A `ChatCompletionAgent` uses `ChatHistoryAgentThread` to manage conversation history. It can be initialized as follows: + +```python +from semantic_kernel.agents import ChatHistoryAgentThread + +thread = ChatHistoryAgentThread( + chat_history: ChatHistory | None = None, + thread_id: str | None = None +) +``` + +Providing a `thread_id` allows continuing existing conversations. Omitting it creates a new thread. Serialization and rehydration of thread state are supported for persistent conversation contexts. + +A complete implementation example: + +```python +import asyncio + +from semantic_kernel.agents import ChatCompletionAgent, ChatHistoryAgentThread +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion + +# Simulate a conversation with the agent +USER_INPUTS = [ + "Hello, I am John Doe.", + "What is your name?", + "What is my name?", +] + + +async def main(): + # 1. Create the agent by specifying the service + agent = ChatCompletionAgent( + service=AzureChatCompletion(), + name="Assistant", + instructions="Answer the user's questions.", + ) + + # 2. Create a thread to hold the conversation + # If no thread is provided, a new thread will be + # created and returned with the initial response + thread: ChatHistoryAgentThread = None + + for user_input in USER_INPUTS: + print(f"# User: {user_input}") + # 3. Invoke the agent for a response + response = await agent.get_response( + messages=user_input, + thread=thread, + ) + print(f"# {response.name}: {response}") + # 4. Store the thread, which allows the agent to + # maintain conversation history across multiple messages. + thread = response.thread + + # 5. Cleanup: Clear the thread + await thread.delete() if thread else None + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### OpenAI Assistant Thread + +The `AzureAssistantAgent` and `OpenAIAssistantAgent` use `AssistantAgentThread` to manage conversation history and context: + +```python +from semantic_kernel.agents import ChatHistoryAgentThread + +thread = AssistantAgentThread( + client: AsyncOpenAI, + thread_id: str | None = None, + messages: Iterable["ThreadCreateMessage"] | NotGiven = NOT_GIVEN, + metadata: dict[str, Any] | NotGiven = NOT_GIVEN, + tool_resources: ToolResources | NotGiven = NOT_GIVEN, +) +``` + +Providing a `thread_id` continues an existing conversation; otherwise, a new thread is created. + +A complete implementation example: + +```python +# Copyright (c) Microsoft. All rights reserved. +import asyncio + +from semantic_kernel.agents import AzureAssistantAgent + + +# Simulate a conversation with the agent +USER_INPUTS = [ + "Why is the sky blue?", + "What is the speed of light?", + "What have we been talking about?", +] + + +async def main(): + # 1. Create the client using Azure OpenAI resources and configuration + client, model = AzureAssistantAgent.setup_resources() + + # 2. Create the assistant on the Azure OpenAI service + definition = await client.beta.assistants.create( + model=model, + instructions="Answer questions about the world in one sentence.", + name="Assistant", + ) + + # 3. Create a Semantic Kernel agent for the Azure OpenAI assistant + agent = AzureAssistantAgent( + client=client, + definition=definition, + ) + + # 4. Create a new thread for use with the assistant + # If no thread is provided, a new thread will be + # created and returned with the initial response + thread = None + + try: + for user_input in USER_INPUTS: + print(f"# User: '{user_input}'") + # 6. Invoke the agent for the current thread and print the response + response = await agent.get_response(messages=user_input, thread=thread) + print(f"# {response.name}: {response}") + thread = response.thread + + finally: + # 7. Clean up the resources + await thread.delete() if thread else None + await agent.client.beta.assistants.delete(assistant_id=agent.id) + + +if __name__ == "__main__": + asyncio.run(main()) + +``` + +## Message Inputs for Agent Invocation + +Previous implementations allowed only a single message input to methods like `get_response(...)`, `invoke(...)`, and `invoke_stream(...)`. We've now updated these methods to support multiple `messages (str | ChatMessageContent | list[str | ChatMessageContent])`. Message inputs need to be passed in with the `messages` keyword argument, such as `agent.get_response(messages="user input")` or `agent.invoke(messages="user input")`. + +Agent invocation methods need updates as follows: + +### Old Way + +```python +response = await agent.get_response(message="some user input", thread=thread) +``` + +### New Way + +```python +response = await agent.get_response(messages=["some initial inputer", "other input"], thread=thread) +``` + +## `AzureAIAgent` + +In Semantic Kernel Python 1.26.0+, `AzureAIAgent` thread creation is now managed via the `AzureAIAgentThread` object, not directly on the client. + +### Old Way + +```python +thread = await client.agents.create_thread() +``` + +### New Way + +```python +from semantic_kernel.agents import AzureAIAgentThread + +thread = AzureAIAgentThread( + client: AIProjectClient, # required + messages: list[ThreadMessageOptions] | None = None, # optional + metadata: dict[str, str] | None = None, # optional + thread_id: str | None = None, # optional + tool_resources: "ToolResources | None" = None, # optional +) +``` + +If no `thread_id` is provided initially, a new thread is created and returned in the agent response. + ## `ChatCompletionAgent` The `ChatCompletionAgent` has been updated to simplify service configuration, plugin handling, and function calling behaviors. Below are the key changes you should consider when migrating. @@ -189,6 +690,9 @@ You can now specify the service directly as part of the agent constructor: #### New Way ```python +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion + agent = ChatCompletionAgent( service=AzureChatCompletion(), name="", @@ -203,6 +707,9 @@ Note: If both a kernel and a service are provided, the service will take precede Previously, you would first add a service to a kernel and then pass the kernel to the agent: ```python +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion + kernel = Kernel() kernel.add_service(AzureChatCompletion()) @@ -220,6 +727,9 @@ Plugins can now be supplied directly through the constructor: #### New Way ```python +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion + agent = ChatCompletionAgent( service=AzureChatCompletion(), name="", @@ -233,6 +743,9 @@ agent = ChatCompletionAgent( Plugins previously had to be added to the kernel separately: ```python +from semantic_kernel.agents import ChatCompletionAgent +from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion + kernel = Kernel() kernel.add_plugin(SamplePlugin()) @@ -249,22 +762,31 @@ Note: Both approaches are valid, but directly specifying plugins simplifies init You now have two ways to invoke the agent. The new method directly retrieves a single response, while the old method supports streaming. -#### New Way (Single Response) +#### New Way (No Conversation Thread/Context) ```python -chat_history = ChatHistory() -chat_history.add_user_message("") -response = await agent.get_response(chat_history) -# response is of type ChatMessageContent +response = await agent.get_response(messages="user input") +# response is of type AgentResponseItem[ChatMessageContent] ``` +Note: if the next response does not use the returned thread, the conversation will use a new thread and thus will not continue with previous context. -#### Old Way (Still Valid) +#### New Way (Single Response with Context) + +```python +thread = ChatHistoryAgentThread() + +for user_input in ["First user input", "Second User Input"]: + response = await agent.get_response(messages=user_input, thread=thread) + # response is of type AgentResponseItem[ChatMessageContent] + thread = response.thread +``` + +#### Old Way (No Longer Valid) ```python chat_history = ChatHistory() chat_history.add_user_message("") -async for response in agent.invoke(chat_history): - # handle response +response = agent.get_response(message="user input", chat_history=chat_history) ``` ### 4. Controlling Function Calling @@ -404,8 +926,20 @@ thread_id = await agent.create_thread() ### New Way ```python -thread = await agent.client.beta.threads.create() -# Use thread.id for the thread_id string +from semantic_kernel.agents AssistantAgentThread, AzureAssistantAgent + +client, model = AzureAssistantAgent.setup_resources() + +# You may create a thread based on an existing thread id +# thread = AssistantAgentThread(client=client, thread_id="existing-thread-id") +# Otherwise, if not specified, a thread will be created during the first invocation +# and returned as part of the response +thread = None + +async for response in agent.invoke(messages="user input", thread=thread): + # handle response + print(response) + thread = response.thread ``` ## 3. Handling Plugins @@ -580,7 +1114,7 @@ await agent.delete() ### New Way ```python await client.files.delete(file_id) -await client.beta.threads.delete(thread.id) +await thread.delete() await client.beta.assistants.delete(agent.id) ``` diff --git a/semantic-kernel/support/migration/toc.yml b/semantic-kernel/support/migration/toc.yml index 1718b55c..aa17fe21 100644 --- a/semantic-kernel/support/migration/toc.yml +++ b/semantic-kernel/support/migration/toc.yml @@ -11,4 +11,6 @@ - name: Kernel Events and Filters Migration href: kernel-events-and-filters-migration.md - name: Agent Framework Release Candidate Migration Guide - href: agent-framework-rc-migration-guide.md \ No newline at end of file + href: agent-framework-rc-migration-guide.md +- name: Vector Store changes - March 2025 + href: vectorstore-march-2025.md \ No newline at end of file diff --git a/semantic-kernel/support/migration/vectorstore-march-2025.md b/semantic-kernel/support/migration/vectorstore-march-2025.md new file mode 100644 index 00000000..dececefe --- /dev/null +++ b/semantic-kernel/support/migration/vectorstore-march-2025.md @@ -0,0 +1,149 @@ +--- +title: Vector Store changes - March 2025 +description: Describes the changes included in the March 2025 Vector Store release and how to migrate +zone_pivot_groups: programming-languages +author: westey-m +ms.topic: conceptual +ms.author: westey +ms.date: 03/06/2025 +ms.service: semantic-kernel +--- +::: zone pivot="programming-language-csharp" + +# Vector Store changes - March 2025 + +## LINQ based filtering + +When doing vector searches it is possible to create a filter (in addition to the vector similarity) +that act on data properties to constrain the list of records matched. + +This filter is changing to support more filtering options. Previously the filter would +have been expressed using a custom `VectorSearchFilter` type, but with this update the filter +would be expressed using LINQ expressions. + +The old filter clause is still preserved in a property called OldFilter, and will be removed in future. + +```csharp +// Before +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { Filter = new VectorSearchFilter().EqualTo(nameof(Glossary.Category), "External Definitions") }); + +// After +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { Filter = g => g.Category == "External Definitions" }); + +// The old filter option is still available +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { OldFilter = new VectorSearchFilter().EqualTo(nameof(Glossary.Category), "External Definitions") }); +``` + +## Target Property Selection for Search + +When doing a vector search, it is possible to choose the vector property that the search should +be executed against. +Previously this was done via an option on the `VectorSearchOptions` class called `VectorPropertyName`. +`VectorPropertyName` was a string that could contain the name of the target property. + +`VectorPropertyName` is being obsoleted in favour of a new property called `VectorProperty`. +`VectorProperty` is an expression that references the required property directly. + +```csharp +// Before +var options = new VectorSearchOptions() { VectorPropertyName = "DescriptionEmbedding" }; + +// After +var options = new VectorSearchOptions() { VectorProperty = r => r.DescriptionEmbedding }; +``` + +Specifying `VectorProperty` will remain optional just like `VectorPropertyName` was optional. +The behavior when not specifying the property name is changing. +Previously if not specifying a target property, and more than one vector property existed on the +data model, the search would target the first available vector property in the schema. + +Since the property which is 'first' can change in many circumstances unrelated to the search code, using this +strategy is risky. We are therefore changing this behavior, so that if there are more than +one vector property, one must be chosen. + +## `VectorSearchOptions` change to generic type + +The `VectorSearchOptions` class is changing to `VectorSearchOptions`, to accomodate the +LINQ based filtering and new property selectors metioned above. + +If you are currently constructing the options class without providing the name of the options class +there will be no change. E.g. `VectorizedSearchAsync(embedding, new() { Top = 5 })`. + +On the other hand if you are using `new` with the type name, you will need to add the record type as a +generic parameter. + +```csharp +// Before +var options = new VectorSearchOptions() { Top = 5 }; + +// After +var options = new VectorSearchOptions() { Top = 5 }; +``` + +## Removal of collection factories in favour of inheritance/decorator pattern + +Each VectorStore implementation allows you to pass a custom factory to use for +constructing collections. This pattern is being removed and the recommended approach +is now to inherit from the VectorStore where you want custom construction and override +the GetCollection method. + +```csharp +// Before +var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + new() + { + VectorStoreCollectionFactory = new CustomQdrantCollectionFactory(productDefinition) + }); + +// After +public class QdrantCustomCollectionVectorStore(QdrantClient qdrantClient) : QdrantVectorStore(qdrantClient) +{ + public override IVectorStoreRecordCollection GetCollection(string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition = null) + { + // custom construction logic... + } +} + +var vectorStore = new QdrantCustomCollectionVectorStore(new QdrantClient("localhost")); +``` + +## Removal of DeleteRecordOptions and UpsertRecordOptions + +The `DeleteRecordOptions` and `UpsertRecordOptions` parameters have been removed from the +`DeleteAsync`, `DeleteBatchAsync`, `UpsertAsync` and `UpsertBatchAsync` methods on the +`IVectorStoreRecordCollection` interface. + +These parameters were all optional and the options classes did not contain any options to set. + +If you were passing these options in the past, you will need to remove these with this update. + +```csharp +// Before +collection.DeleteAsync("mykey", new DeleteRecordOptions(), cancellationToken); + +// After +collection.DeleteAsync("mykey", cancellationToken); +``` + +::: zone-end +::: zone pivot="programming-language-python" + +## Not Applicable + +These changes are currently only applicable in C# + +::: zone-end +::: zone pivot="programming-language-java" + +## Coming soon + +These changes are currently only applicable in C# + +::: zone-end