Figure out an online testing strategy

[woodruffw]

Our online audits need/deserve snapshot/acceptance tests too! These should always run in CI, and should be optionally runnable by local users who are willing to set GH_TOKEN or similar in their environment.

One idea: we could mark online tests with #[ignore], and then run all non-ignored tests in CI.

Another idea: we could define an online_tests feature and do something like #[cfg_attr(not(feature = "online_tests"), ignore)] on each test.

Not sure if either of these is good/best practice, though.

[ubiratansoares]

Perhaps Mock Servers could be an option here!

From the experience with JVM projects I can definitely recommend tools like WireMock and TestContainers, but eventually we can go simple for Rust and use httpmock.rs to mock what we need from Github API.

A couple of things to consider for this solution:

Could be better

• the existing Github HTTP client we have in zizmormust retrieve the Github API URL dynamically at testing time, since it will be one provided by the Mock Server (and in general it may change per instance, e.g mapped to different TCP ports)
• We can do that either by a convention env var or CLI argument (and even protect this behaviour with a feature flag as well)
• only for integration tests (I think)
• Mock data can always become out of sync with the schema of API responses

Looks great

• the value provided to GITHUB_TOKEN at testing time won't matter at all 😄
• httpmock.rs a has record/replay feature which helps to setup mock data easily
• we can setup Mock data using real world OSS projects
• httpmock.rs supports Mock data as YAML files, which eventually enables automation on top of it
• no Docker or other Container runtime required
• works the same at local machine and CI

Wdyt?

[woodruffw]

I like this idea! However, I don't think we should start there: IMO we do need some real full-online integration tests as an end-to-end matter, since mocking is always an approximation of program behavior.

IMO we should start here with some actually-online tests, and then evaluate httpmock.rs (TIL, very cool!) for additional coverage, particularly of error/exceptional cases 🙂