You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.
There's a few challenges to tackle when handling this though:
Most of the test suite is not architected to run the wasmtime CLI or the libwasmtime.so C API. Some of tests/all/cli_tests.rs runs the wasmtime executable, but the executable is produced as part of cargo test and isn't a released artifact. The C API is only tested through the C examples (so very lightly).
Testing artifacts might require a different command to execute or new infrastructure other than cargo test. It's unclear what this infrastructure would look like.
Testing "all possible configurations" isn't feasible, even with the generous concurrency that we already have. For example we can't test the matrix of {platform, opt/debug, debug-asserts/no, crate} in CI as that would be too much.
We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:
We should probably have one CI job test with --release to test with optimizations (just one platform though, probably not all of them)
We should ideally run at least a "smoke test" for released artifacts.
In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.
There's a few challenges to tackle when handling this though:
wasmtime
CLI or thelibwasmtime.so
C API. Some oftests/all/cli_tests.rs
runs thewasmtime
executable, but the executable is produced as part ofcargo test
and isn't a released artifact. The C API is only tested through the C examples (so very lightly).cargo test
. It's unclear what this infrastructure would look like.We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:
--release
to test with optimizations (just one platform though, probably not all of them)cc @tschneidereit
The text was updated successfully, but these errors were encountered: