AgentCI Client Config

Define evaluations and framework configurations for AI agent applications using simple TOML files.

📚 Full Documentation | 🚀 Getting Started | 📖 TOML Schema Guides

What is This?

AgentCI Client Config provides a TOML-based configuration format for:

• Evaluations: Test cases for AI agents and tools with support for accuracy, performance, consistency, and safety testing
• Framework Configurations: Patterns for discovering agents and tools in popular AI frameworks (LangChain, LlamaIndex, Pydantic AI, OpenAI, Google ADK, Agno)

Quick Example

Create evaluation configs in .agentci/evals/:

# .agentci/evals/test_accuracy.toml
[eval]
description = "Test that the agent responds with correct information"
type = "accuracy"

[eval.targets]
agents = ["my_agent"]

[[eval.cases]]
prompt = "What is the capital of France?"
expected.exact = "Paris"

Create framework configs in .agentci/frameworks/:

# .agentci/frameworks/my_framework.toml
[framework]
name = "my-framework"
dependencies = ["my-framework"]

[[agents]]
path = "my_framework.Agent"
args.model = "llm"
args.prompt = "system_prompt"
execution.method = "run"
execution.args.prompt = "user_input"

Installation

pip install agentci-client-config

Documentation

For complete TOML schema documentation and guides:

• Getting Started - Project setup and first configs
• Evaluation Schema - Complete guide to evaluation TOML format
• Framework Schema - Complete guide to framework TOML format
• Python API - Optional programmatic usage

Features

Evaluations

• Six evaluation types: accuracy, performance, consistency, safety, llm, custom
• Flexible matching: exact, contains, regex, semantic similarity
• Schema validation: Validate structured JSON outputs
• Tool call validation: Verify correct tool usage
• Multiple iterations: Run tests multiple times for consistency

Frameworks

• Built-in support: LangChain, LlamaIndex, Pydantic AI, OpenAI Agents, Google ADK, Agno
• Custom frameworks: Define your own discovery patterns
• Agent discovery: Map framework parameters to standard fields
• Tool discovery: Configure tool types (decorator, function, class, constructor)
• Execution config: Define how to run agents and tools

Directory Structure

your-project/
├── .agentci/
│   ├── evals/              # Evaluation configurations
│   │   ├── accuracy.toml
│   │   ├── performance.toml
│   │   └── safety.toml
│   └── frameworks/         # Framework configurations (optional)
│       └── custom.toml
├── src/
└── tests/

License

MIT