Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move towards "test the exact release artifacts" in CI #9881

Open
alexcrichton opened this issue Dec 20, 2024 · 0 comments
Open

Move towards "test the exact release artifacts" in CI #9881

alexcrichton opened this issue Dec 20, 2024 · 0 comments

Comments

@alexcrichton
Copy link
Member

In yesterday's Wasmtime meeting we had a discussion about testing, CI, and the artifacts that we produce. Currently the actual released artifacts are not actually tested in CI at all, they're simply produced. This isn't a great situation and ideally we would test the exact artifacts that CI produces rather than a different build configuration.

There's a few challenges to tackle when handling this though:

  • Most of the test suite is not architected to run the wasmtime CLI or the libwasmtime.so C API. Some of tests/all/cli_tests.rs runs the wasmtime executable, but the executable is produced as part of cargo test and isn't a released artifact. The C API is only tested through the C examples (so very lightly).
  • Testing artifacts might require a different command to execute or new infrastructure other than cargo test. It's unclear what this infrastructure would look like.
  • Testing "all possible configurations" isn't feasible, even with the generous concurrency that we already have. For example we can't test the matrix of {platform, opt/debug, debug-asserts/no, crate} in CI as that would be too much.

We didn't reach many firm conclusions in the meeting but I wanted to open a follow-up issue to track further progress. A few points we did identify were:

  • We should probably have one CI job test with --release to test with optimizations (just one platform though, probably not all of them)
  • We should ideally run at least a "smoke test" for released artifacts.

cc @tschneidereit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant