we need E2E/Integration tests

[lucasbalieiro]

The application has grown to a point where it is becoming difficult to keep the entire context in my head. We now integrate with several operating systems, multiple Bitcoin networks, and different scenarios where Bitcoin Core may or may not be present.

Reviewing PRs has also become increasingly cumbersome because we still rely heavily on manual testing. It is often unclear whether a change is unintentionally breaking behavior in another part of the system.

The workflow itself is quite complex. We depend on Docker, and there are many failure scenarios to account for, such as services being unavailable and ensuring the UI reacts as expected in those situations.

I do not yet know how to approach this or have time to properly plan it right now. I am only writing this down as a reminder to revisit later.

[lucasbalieiro]

I've used tools like Cypress and Playwright in the past for E2E testing to validate critical user flows and UI behavior, such as:

• onboarding/setup
• error banners
• enabled/disabled actions
• recovery after failures

One open question is how much we should rely on mocks.

For some scenarios, mocking could be useful (RPC responses from Core, captured container logs used by diagnostics, ASIC devices such as Bitaxe, etc.). This would reduce CI resource usage and speed up tests. The downside is the maintenance burden, since mocks need to evolve together with the real applications.

My current inclination would be to prefer real dependencies whenever possible (real Core instance and real managed containers), and only use mocks for scenarios that are difficult or expensive to reproduce. The tests would be slower, but they would provide much higher confidence that integrations still behave correctly.

We could leverage Playwright's own test harnesses and potentially build a small abstraction around environment setup, allowing tests to define scenarios such as:

testEnv.stopCore();
testEnv.startCore();
testEnv.dropContainer();
testEnv.disableDocker();

A lot of the testing does not even require third party services. The wizard process, form validation, and general UI behavior can likely be covered with straightforward Playwright scripts.

Some initial high value scenarios to automate:

• Are we generating the correct configuration files from the wizard flow?
• Are diagnostics correctly surfacing the large error banners?
• Does the UI properly react when Docker is unavailable at startup?
• Are forms validating inputs correctly and surfacing user friendly errors (e.g. authority pubkey validation, addresses etc)?
• Does the UI react correctly when Core or managed containers become unavailable?
• The different scenarios are correctly spawning up the correct containers?

These tests are intentionally higher level. The goal is not to validate every network message or internal interaction, but rather to verify that the application reacts correctly to changes in the environment and presents the expected state to the user.