feat(testing): add MVP TOML DSL adapter and test dsl CLI command#1555
Open
AdityaShome wants to merge 12 commits intomofa-org:mainfrom
Open
feat(testing): add MVP TOML DSL adapter and test dsl CLI command#1555AdityaShome wants to merge 12 commits intomofa-org:mainfrom
AdityaShome wants to merge 12 commits intomofa-org:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is rebased on #1447 and adds an MVP TOML based testing DSL to the root level
tests/crate and exposes it through a newmofa test-dslCLI command.The DSL is intentionally narrow. It is a thin adapter over the existing testing runner and report stack, so contributors can define and execute simple agent test cases without introducing a separate DSL execution framework.
This PR also adds optional report-file output for DSL runs, reusing the existing report types and formatters from the
tests/crate.What Changed
tests/crate for simple agent test casesmofa test-dsl <file>to execute TOML test cases through the existing testing runner--report-outand--report-format {json,text}Files Changed
Core Files
crates/mofa-cli/src/cli.rstest-dslsubcommand.--report-outand--report-formatoptions.TestDslReportFormatwithjsonandtext.crates/mofa-cli/src/commands/mod.rstest_dslcommand module.crates/mofa-cli/src/commands/test_dsl.rsmofa test-dslcommand.tests/crate, prints a text/JSON summary, and optionally writes a report file using the existing report formatters.crates/mofa-cli/src/main.rstest-dslsubcommand into CLI dispatch.tests/src/dsl.rsAgentTestRunner, and validates simple assertions.tests/src/lib.rstests/crate.tests/tests/dsl_tests.rsSupporting Files
crates/mofa-cli/tests/test_dsl_integration_tests.rs: CLI integration coverage for DSL execution, JSON stdout output, and JSON/text report generation.tests/examples/simple_agent.toml: Minimal example DSL case with a prompt, mocked LLM response, and text assertion.tests/examples/bootstrap_agent.toml: DSL example covering bootstrap file loading and prompt bootstrap integration.tests/examples/tool_agent.toml: Tool-aware example DSL case with agent metadata, mock tool setup, stepped LLM tool calls, and assertions.DSL Scope
The initial DSL supports:
promptorinputCLI Usage
Examples:
The command prints a run summary to stdout and can optionally write a structured report file in
jsonortextformat.Flow
flowchart TD A[TOML DSL file] --> B[mofa test-dsl] B --> C[Parse DSL into TestCaseDsl] C --> D[Configure existing AgentTestRunner] D --> E[Apply bootstrap files] D --> F[Register mock tools] D --> G[Queue deterministic LLM responses or steps] E --> H[Execute test case] F --> H G --> H H --> I[Collect run result and metadata] I --> J[Apply DSL assertions] J --> K[Print CLI summary] J --> L[Build TestReport] L --> M[Optional report file<br/>json or text]Example Output
Dependency Changes
tomldependency for shared TOML parsingtomlto thetests/cratetests/crate as a dependency ofmofa-cliTests
cargo test -p mofa-testing --test dsl_testscargo test -p mofa-cli --test test_dsl_integration_testsNotes
This keeps the DSL intentionally small and runner backed for now, so future DSL expansion can build on the same execution and reporting path.