Skip to content

feat(testing): add MVP TOML DSL adapter and test dsl CLI command#1555

Open
AdityaShome wants to merge 12 commits intomofa-org:mainfrom
AdityaShome:testing-dsl-adapter
Open

feat(testing): add MVP TOML DSL adapter and test dsl CLI command#1555
AdityaShome wants to merge 12 commits intomofa-org:mainfrom
AdityaShome:testing-dsl-adapter

Conversation

@AdityaShome
Copy link
Copy Markdown
Contributor

Summary

This PR is rebased on #1447 and adds an MVP TOML based testing DSL to the root level tests/ crate and exposes it through a new mofa test-dsl CLI command.

The DSL is intentionally narrow. It is a thin adapter over the existing testing runner and report stack, so contributors can define and execute simple agent test cases without introducing a separate DSL execution framework.

This PR also adds optional report-file output for DSL runs, reusing the existing report types and formatters from the tests/ crate.

What Changed

  • Added a minimal TOML DSL in the tests/ crate for simple agent test cases
  • Added mofa test-dsl <file> to execute TOML test cases through the existing testing runner
  • Added optional report output for DSL runs via --report-out and --report-format {json,text}
  • Added example DSL files and focused unit/integration coverage

Files Changed

Core Files

crates/mofa-cli/src/cli.rs

  • Added the test-dsl subcommand.
  • Added --report-out and --report-format options.
  • Added TestDslReportFormat with json and text.

crates/mofa-cli/src/commands/mod.rs

  • Exported the new test_dsl command module.

crates/mofa-cli/src/commands/test_dsl.rs

  • Implemented the mofa test-dsl command.
  • Loads a TOML case, executes it through the tests/ crate, prints a text/JSON summary, and optionally writes a report file using the existing report formatters.

crates/mofa-cli/src/main.rs

  • Wired the test-dsl subcommand into CLI dispatch.

tests/src/dsl.rs

  • Added the minimal TOML DSL adapter.
  • Defines the schema, parses TOML files, configures AgentTestRunner, and validates simple assertions.

tests/src/lib.rs

  • Exported the new DSL types and helpers from the tests/ crate.

tests/tests/dsl_tests.rs

  • Added coverage for simple DSL execution, bootstrap file support, and tool-backed runs.

Supporting Files

crates/mofa-cli/tests/test_dsl_integration_tests.rs: CLI integration coverage for DSL execution, JSON stdout output, and JSON/text report generation.

tests/examples/simple_agent.toml: Minimal example DSL case with a prompt, mocked LLM response, and text assertion.

tests/examples/bootstrap_agent.toml: DSL example covering bootstrap file loading and prompt bootstrap integration.

tests/examples/tool_agent.toml: Tool-aware example DSL case with agent metadata, mock tool setup, stepped LLM tool calls, and assertions.

DSL Scope

The initial DSL supports:

  • prompt or input
  • bootstrap files
  • agent name/description
  • mock tool registration with fixed results
  • deterministic LLM responses
  • simple stepped LLM flows, including tool calls
  • lightweight assertions such as output containment and expected tool call

CLI Usage

Examples:

mofa test-dsl tests/examples/simple_agent.toml
mofa --output-format json test-dsl tests/examples/tool_agent.toml
mofa test-dsl tests/examples/simple_agent.toml --report-out dsl-report.json --report-format json

The command prints a run summary to stdout and can optionally write a structured report file in json or text format.

Flow

flowchart TD
    A[TOML DSL file] --> B[mofa test-dsl]
    B --> C[Parse DSL into TestCaseDsl]
    C --> D[Configure existing AgentTestRunner]
    D --> E[Apply bootstrap files]
    D --> F[Register mock tools]
    D --> G[Queue deterministic LLM responses or steps]
    E --> H[Execute test case]
    F --> H
    G --> H
    H --> I[Collect run result and metadata]
    I --> J[Apply DSL assertions]
    J --> K[Print CLI summary]
    J --> L[Build TestReport]
    L --> M[Optional report file<br/>json or text]
Loading

Example Output

mofa test-dsl tests/examples/simple_agent.toml
case: simple_agent_run
status: passed
output: hello from DSL
duration_ms: 0

Dependency Changes

  • Added workspace-level toml dependency for shared TOML parsing
  • Added toml to the tests/ crate
  • Added the tests/ crate as a dependency of mofa-cli

Tests

  • cargo test -p mofa-testing --test dsl_tests
  • cargo test -p mofa-cli --test test_dsl_integration_tests

Notes

This keeps the DSL intentionally small and runner backed for now, so future DSL expansion can build on the same execution and reporting path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant