How to evaluate your generative AI application using Azure AI Foundry. Whether you're assessing single-turn or multi-turn conversations, Azure AI Foundry provides tools for evaluating model performance and safety.
For more details instruction see the Azure AI Foundry Documentation
Here are the steps to get started:
Prerequisites
- A test dataset in either CSV or JSON format.
- A deployed generative AI model (such as Phi-3, GPT 3.5, GPT 4, or Davinci models).
- A runtime with a compute instance to run the evaluation.
Azure AI Foundry allows you to evaluate both single-turn and complex, multi-turn conversations. For Retrieval Augmented Generation (RAG) scenarios, where the model is grounded in specific data, you can assess performance using built-in evaluation metrics. Additionally, you can evaluate general single-turn question answering scenarios (non-RAG).
From the Azure AI Foundry UI, navigate to either the Evaluate page or the Prompt Flow page. Follow the evaluation creation wizard to set up an evaluation run. Provide an optional name for your evaluation. Select the scenario that aligns with your application's objectives. Choose one or more evaluation metrics to assess the model's output.
For greater flexibility, you can establish a custom evaluation flow. Customize the evaluation process based on your specific requirements.
After running the evaluation, log, view, and analyze detailed evaluation metrics in Azure AI Foundry. Gain insights into your application's capabilities and limitations.
Note Azure AI Foundry is currently in public preview, so use it for experimentation and development purposes. For production workloads, consider other options. Explore the official AI Foundry documentation for more details and step-by-step instructions.