Dev: Explore Reducing Test Run Time

[lread]

Currently

Tests can take a very long while to run.
This is painful when developing Etaoin.
My computer is getting long in the tooth, but it just finished a test run against chrome and firefox in ~24m.

Ideas

@dgr and I chatted about the expense of creating a WebDriver process for each test.
This is great in that it provides complete isolation for each test but is time-consuming.

Questions:

1. Can we reuse a WebDriver process and/or session across tests?

In some cases, the answer will be no. Sometimes a WebDriver test launches the WebDriver with specific arguments (this is not the case for api tests, but is for other tests). Other times, a test will affect the session state, with, for example, cookies. And what should we do if a test fails? Probably throw away our WebDriver process and start fresh.

But, often, I expect a simple page refresh would be enough to clean the slate.

2. Can we parallelize when testing against multiple browsers?

Maybe. But the complexity might not be worth it.
Note that CI is parallelized by running one browser per job.

Next Steps

Experiment with the above. Measure the impact on tests and the time it takes to run them.

[dgr]

A few random thoughts:

1. There are lots of places where there are delays in the test code. Some of that might have been necessary at some point (there are some Safari-specific delays in the input testing code, for instance) but may not be necessary any more with the most recent browser versions. Further, there are some wait tests that seem to wait for many seconds for specific text to become available. Could those delays be shortened up to a second or less and still test the same thing?
2. Could you use tabs or windows within the same browser instance to try each test? Then, you would pay the startup cost once for each browser and then whatever the new tab or new window cost is for each test. I have to think the cost of a new tab is less than the cost of restarting the whole browser each time.
3. Parallelize things? Spawn a separate thread for each browser?

[lread]

Main thought: The world of WebDrivers and browsers is already flaky, and we need to avoid making it more so.

1. Yeah, Safari is the worst and weirdest of the bunch. There is also CI vs dev box. CI on Windows and macOS will typically run much more slowly and flakily than on a dev box.

2. I don't know if tabs or windows provide more isolation. In most cases, a simple go to the test page could be enough for most tests. Tests that need more isolation could be treated separately.

3. This one is trickier... we'd still have to show test output coherently. But this is the basic strategy we took on CI with separate jobs for each browser.

Another thing might be to explore running headless, Safari still does not support running headless, but things might have changed for other browsers. Is running headless equivalent to headed browser behaviour these days? (It did not used to be) Is it faster? It certainly is less disruptive in that it doesn't take over your dev box for the duration of the test.