Library Testing Discussion

[kyleboe]

In the context of work happening on #536 this concern was raised by @Kraigie in discord:

I think my biggest worry is trying to be sure that all of the changes are actually correct. We don't have formal testing for and of the API module really, and I'm not sure it's feasible to go through and actually test all of them. It feels like our strategy would basically be push it out and see what people complain about which is a bit weird IMO

I agree with this sentiment wholeheartedly and there needs to be some discussion around how to properly "formally" test this library. This is also not the first time this concern has been raised (#250). In an effort to remedy this issue holistically, I have outlined some potential solutions below.

I have given each suggestion a rough approximation of Effort involved to accomplish the task as well as Confidence the solution provides. For this, I'll define confidence as the percentage of systems involved being tested, whether directly or indirectly. For example, stubbing out massive swaths of function calls gives us a lower degree of confidence relative to exercising the complete, end-to-end network request/response lifecycle.

DiscordMock

This approach borrows from the Stripe ecosystem having a small, portable server that mocks the entire API.

https://github.com/stripe/stripe-mock

This would give us a solid foundation on which to completely exercise the end-to-end lifecycle of a request and subsequent response. We could optionally structure DiscordMock in such a way that we could have data exist throughout the lifecycle of a test (i.e. if you create a guild in one call, you can reference that guild in the next call, all in the context of a single test). Or go the route of StripeMock and let each request be idempotent (quote from their Readme below):

stripe-mock is meant for basic sanity checks. We use it in the test suites of our server-side SDKs to help validate that the SDK hits the right URL and sends the right parameters.

Effort: 5/5

Confidence: 5/5

Bypass

Bypass is effectively a network-level mocking library that spins up a server to return a predefined response. This allows us to test the network but opens the door to us potentially lying to ourselves by creating a stub that does not align to changes in the real-world Discord API. Although, this is an issue with mocking at any level.

Effort: 4/5

Confidence: 4/5

Stubbing

We could find a way to stub API calls at a functional level. Moving this direction requires a non-trivial amount of refactoring because of how tightly coupled the library is to gun. We will need to have forks in the code paths specific to the test environment which could get messy if not implemented thoughtfully/consistently.

Effort: 3/5

Confidence: 3/5

---

Those are just the first three that popped into my head so if anyone sees another path forward, let's discuss below.