feat: custom test harness #817

lima-limon-inc · 2025-12-11T22:29:01Z

Note: this PR is mainly intended to be an exploration on what a custom test harness for miden could look like. Comments/feeback more than welcome!

This PR implements a custom test harness for miden rust code. The custom harness itself is comprised of two crates:

miden-test-harness-lib: Library which serves as a small wrapper to libtest_mimic
miden-test-harness-macros: Crate containing the #[miden_test] and #[miden_test_block] macros; which serve as miden-test specific equivalents for #[test] and mod tests respectively.

#[miden_test] adds every function to an inventory list. Additionally, if the function receives a Package as argument, it will load the generated Package into a variable automatically; if no argument is passed, the Package isn't loaded.

The counter-contract example was updated to showcase what these tests would look like.

Additionally, a new subcommand was added to cargo miden called test, which calls cargo miden build to generate the corresponding .masp Package and then calls cargo test with the required flags.

Limitation/Places to go:

I was not getting code completion inside the test functions themselves (IDE: GNU Emacs 30.1, rust-analyzer: rust-analyzer 1.91.0 (f8297e35 2025-10-28). This, I believe is related to greenhat's comments
I needed to omit the AccountComponent rust code when running cargo test in order for the test suite to run, if not, cargo would simply panic. I believe this breakage came from the different targets that are required: cargo build targetting wasm, whilst cargo test targetting the native architecture. Hence, I decided to move the AccountComponent code into a separate module that was only compiled when not using the test profile. However, I don't believe this setup is ideal/final; only a temporary fix.
Finally, since the MockChainBuilder is so versatile and has a lot of customization options, I struggled to automatically generate a MockChain and thus the test ended up having a lot of boilerplate. Maybe defining sane defaults would aid in passing in a pre-built MockChain and thus skipping this boilerplate.

Would love to hear your ideas!

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

greenhat · 2025-12-15T06:23:21Z

Regarding cargo miden test. We need VS Code test infra working with our harness. I mean
, etc.

It means that cargo test should work with our harness.

/cc @bitwalker

greenhat · 2025-12-15T06:27:47Z

Also, rust-lang/rust-analyzer#16662 test explorer is coming. Which we'd want to support as well.

EDIT: linking the custom test harness integration issue - rust-lang/rust-analyzer#21259

…-lib Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

…the rest untouched Additionally, renamed the function from load_account and added documentation. Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

bitwalker

Nice work! I have left various notes and requested some changes, but I think this is a great starting point for experimenting further with this approach.

I think with some of the changes I suggested, we can get the test discovery working (per @greenhat's request), simply by ensuring that the test code is always emitted, just gated behind #[cfg(test)]. This allows rust-analyzer to see the tests. Additional work is likely still needed to enable integration with rust-analyzer's test lens (e.g. making the test runnable from your IDE), but the basic discovery should work at least.

bitwalker · 2025-12-30T14:43:00Z

test-harness/test-harness-lib/Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "miden-test-harness-lib"


Let's call this crate just miden-test-harness

bitwalker · 2025-12-30T14:48:14Z

test-harness/test-harness-macros/src/lib.rs

+    };
+    input_module.content.as_mut().unwrap().1.insert(0, internal_use);
+
+    let module = if is_test() {


Why not just emit the module annotated with #[cfg(test)]? That should accomplish the same goal, no? It isn't 100% clear why this would need to be facilitated via feature flags on the macro crate (which when you take into account Cargo's feature flag unification approach, could lead to test modules being emitted even when they aren't expected - not likely, but one downside of using features for this sort of thing).

bitwalker · 2025-12-30T14:50:45Z

test-harness/test-harness-macros/src/lib.rs

+            use miden_test_harness_lib;
+
+            fn main() {
+                let args = miden_test_harness_lib::MidenTestArguments::from_args();


I'd always qualify paths absolutely, e.g. ::miden_test_harness_lib::MidenTestArguments::from_args(), this ensures that the path resolves from the crate root, ignoring any imports in the current module. Not a super big problem in this case, but still good practice for proc macros like this.

bitwalker · 2025-12-30T14:53:08Z

test-harness/test-harness-macros/src/lib.rs

+    };
+
+    // This env var is set by `cargo miden test`.
+    let package_path = std::env::var("CREATED_PACKAGE").unwrap();


We should namespace all env vars that we use, e.g. CARGO_MIDEN_CREATED_PACKAGE (though I think it would probably be better to call it CARGO_MIDEN_TEST_PACKAGE_PATH to make it clearer what defines the variable, and what the value is expected to be.

bitwalker · 2025-12-30T14:54:58Z

test-harness/test-harness-macros/src/lib.rs

+}
+
+#[proc_macro_attribute]
+pub fn miden_test_block(


Let's name this attribute #[miden_test_suite], since that is effectively what it is used to define.

bitwalker · 2025-12-30T15:09:14Z

test-harness/test-harness-lib/src/mod.rs

This crate should depend on miden-test-harness-macros, and re-export its macros from the crate root, e.g.:

pub use miden_test_harness_macros::{miden_test, miden_test_suite};

Then, downstream crates need only depend on miden_test_harness

bitwalker · 2025-12-30T15:11:38Z

examples/counter-contract/src/lib.rs

-    /// Storage map holding the counter value.
-    #[storage(slot(0), description = "counter contract storage map")]
-    count_map: StorageMap,
+#[cfg(not(test))]


Instead of gating this behind #[cfg(not(test))], it would make more sense to gate this behind #[cfg(target_family = "wasm")] - our test suite will never be built for the Wasm target, and even if it was, then the compilation error you ran into wouldn't be an issue anyway.

bitwalker · 2025-12-30T15:12:40Z

examples/counter-contract/src/lib.rs

 // Do not link against libstd (i.e. anything defined in `std::`)
-#![no_std]
-#![feature(alloc_error_handler)]
+#![cfg_attr(not(test), no_std)]


We're not actually dependent on std for the code we emit via #[miden_test]/#[miden_test_suite] AFAICT, so do we actually need to make this conditionally compiled?

bitwalker · 2025-12-30T15:14:52Z

examples/counter-contract/src/lib.rs

-    pub fn get_count(&self) -> Felt {
-        let key = Word::from([felt!(0), felt!(0), felt!(0), felt!(1)]);
-        self.count_map.get(&key)
+#[miden_test_block]


I believe test discovery will work if instead of #[miden_test_suite] stripping out the decorated module entirely, it instead annotated it with #[cfg(test)]. I mentioned in my other comment that this would also remove the need for the test_flag feature in miden_test_harness_macros, but the big win here is test discovery.

bitwalker · 2025-12-30T15:17:06Z

examples/counter-contract/src/lib.rs

+    // printed.
+    #[miden_test]
+    fn bar(bar: Package) {
+        std::dbg!(&bar);


You can support this even in #[no_std] crates by simply adding #[cfg(test)] extern crate std; somewhere in the module (conventionally, at the top near the #![no_std] attribute itself, but I believe it can go anywhere (I haven't experimented with that though).

greenhat

Nice work!

The list of showstoppers that we need to solve boils down to:

Working code completion inside the test functions themselves;
Running tests via the lens in vscode(any IDE using rust-analyzer);

These can be addressed in the subsequent PRs (create gh issues). My only request for this PR is to not change the examples/counter-contract.

greenhat · 2026-01-01T06:37:05Z

examples/counter-contract/src/lib.rs

-    #[storage(slot(0), description = "counter contract storage map")]
-    count_map: StorageMap,
+#[cfg(not(test))]
+mod component {


Since this is a user-facing example (i.e. used in cargo miden example) let's copy it to tests/rust-apps-wasm/rust-sdk/ where our internal test examples live.

I'd actually prefer we start placing these integration test examples under a new directory, tests/examples. We've had a longstanding TODO to re-organize and cleanup the integration tests directory structure and module organization, and we may as well start doing that with new code.

greenhat · 2026-01-01T06:49:38Z

tools/cargo-miden/src/commands/test.rs

+
+        let Some(CommandOutput::BuildCommandOutput {
+            output: BuildOutput::Masm { artifact_path },
+        }) = build.exec(OutputType::Masm)?


We discussed that cargo miden test should be just a dumb cargo test alias, nothing more. Otherwise, the rust-analyzer lens and test discovery would not work. Unless vscode calls cargo miden test when the test is started via lens.

That isn't actually true for discovery at least - the test discovery relies purely on analysis of the macro-expanded code, which with the changes I suggested in my review, will work just like any standard test module.

That said, the rust-analyzer test lens is a slightly different story - I haven't looked at how it is implemented, so I don't know how specific it is about how the test gets run.

I agree with the general point though, which is that ideally we can simply make cargo miden test an alias of cargo test, possibly with additional options to allow for us to provide additional Miden-specific functionality when running tests via cargo miden, but hopefully we won't have to require executing tests using cargo miden test, to make IDE integration trivial.

I agree. The point I wanted to make was that we cannot rely on the cargo miden test to compile the package under test. The test harness should compile the package. We could do it in the main function generated by the miden_test_block macro.

The one tricky detail there is going to be whether we will have enough information available when the test binary is built to be able to invoke cargo miden build. I think we should be good, but not 100% sure. I'm guessing it would be enough to bake in the build target name when generating main? Might have to play with it a bit and see what works and what doesn't.

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch 4 times, most recently from 292e2be to 22f5f5b Compare December 12, 2025 20:08

feat: add cargo miden test command

0b16290

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from 22f5f5b to 70fd1f7 Compare December 12, 2025 20:40

lima-limon-inc added 3 commits December 12, 2025 17:55

feat: custom test harness initial commit

ab6a99d

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: add test-harness library to Cargo.toml

5932270

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: update counter-contract example with #[miden-test]

24d1be0

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from 70fd1f7 to 1db9ca0 Compare December 12, 2025 22:16

lima-limon-inc added 3 commits December 16, 2025 16:19

feat: re-export miden-mast-package and miden-objects via test-harness…

a10c5a9

…-lib Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: use custom test harness

519ba8f

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: add miden-testing to test-harness

5bdf3de

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from 1db9ca0 to 5bdf3de Compare December 16, 2025 19:19

lima-limon-inc added 2 commits December 16, 2025 16:22

nit: uncomment test that fails at compile time

f7a8b4d

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

refactor: split miden-test-harness-lib into 2 crates

df9650a

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from 31572ad to df9650a Compare December 16, 2025 20:03

lima-limon-inc added 9 commits December 16, 2025 17:31

feat: add Context struct

2891eb6

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: get_binding_and_type helper function

9c28ba8

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

refactor: load_package now consumes 'Package' related tokens, leaves …

2d67fc2

…the rest untouched Additionally, renamed the function from load_account and added documentation. Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: add load_mock_chain function

5a321a2

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

refactor: process_arguments function to generalize argument parsing

3cd4676

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: add MockChainBuilder to foo test

d61075b

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

fix: add mut to mock chain builder

e1183f2

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: add function with MockChain + Package

c52c16b

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

feat: update Cargo.lock

4056270

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from c620983 to 4056270 Compare December 17, 2025 21:50

feat: remove Context struct

cc5ea76

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc force-pushed the fabrizioorsi/custom-test-harness branch from 6605f89 to cc5ea76 Compare December 18, 2025 16:33

feat: document tests

b4dd9a0

Signed-off-by: Tomas Fabrizio Orsi <[email protected]>

lima-limon-inc marked this pull request as ready for review December 18, 2025 19:09

greenhat self-requested a review December 22, 2025 14:00

bitwalker requested changes Dec 30, 2025

View reviewed changes

greenhat requested changes Jan 1, 2026

View reviewed changes

feat: custom test harness #817

Are you sure you want to change the base?

feat: custom test harness #817

Uh oh!

Conversation

lima-limon-inc commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greenhat commented Dec 15, 2025

Uh oh!

greenhat commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bitwalker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greenhat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lima-limon-inc commented Dec 11, 2025 •

edited

Loading

greenhat commented Dec 15, 2025 •

edited

Loading