RFC: Integration Tests -- How To?

[rotdrop]

Just some brain-storming: this app adds means to obtain the second authentication factor via various service providers. It would be nice to have some "integration tests" -- ideally for all service providers. How could this be done? As most of the service providers require some sort of registration I suppose that this would require a test-account for each service provider (Signal, various SMS-Provides, Ouch WhatsApp etc.).

Perhaps one could ask for sponsoring? Concerning Signal one just needs any telephone number. For other services it might be different.

It would be cool to be able to perform real usage tests against the set of the currently -- perhaps -- supported providers.

Just writing this as an RFC (Request For Comments).

[vitormattos]

One possible approach is to use unit tests, generating mocks of API responses. The LibreSign app has some tests similar to this, which I've done using the project https://github.com/donatj/mock-webserver. I'll think about this further and perhaps create a pull request as a proof of concept.

[rotdrop]

One possible approach is to use unit tests, generating mocks of API responses. The LibreSign app has some tests similar to this, which I've done using the project https://github.com/donatj/mock-webserver. I'll think about this further and perhaps create a pull request as a proof of concept.

Yes, but this adds another bunch of things that need to be kept in sync with the numerous providers. It also does not cover changes in the API which may or may not be announced by the providers. I think that "real integration tests" which access the currently implemented APIs of the numerous providers are easier to maintain and more likely to catch errors as this would be closer to the real-life usage.

[vitormattos]

Real integration tests have some complexities because they depend on managing access credentials. This can create a security issue, as it will allow messages to be sent by multiple providers by anyone with permission to create a pull request that automatically executes CI. Another point is that they are highly susceptible to failures because the external service may not be working for some reason.

If we assume that the code is compatible with the provider's version at the time of the code creation, mocking may not be a problem but a solution to signalize that the implementation respect the API contract. New versions of the providers will certainly break (or are already broken, as there's no way to validate all existing ones) and at this case, opening an issue could be easier and safe.