Update README

ONSdigital · Nov 27, 2024 · e0a271f · e0a271f
1 parent 687e8b7
commit e0a271f
Show file tree

Hide file tree

Showing 2 changed files with 107 additions and 34 deletions.
diff --git a/functional_tests/README.md b/functional_tests/README.md
@@ -1,5 +1,29 @@
 # Functional Tests
 
+<!-- TOC -->
+* [Structure](#structure)
+* [Dependencies](#dependencies)
+  * [App Instance For Test Development](#app-instance-for-test-development)
+  * [Clearing and Initialising the Functional Test Development Database](#clearing-and-initialising-the-functional-test-development-database)
+* [Running the Tests](#running-the-tests)
+  * [Playwright Options](#playwright-options)
+* [Viewing Failure Traces](#viewing-failure-traces)
+  * [Viewing the Failure Trace from GitHub Actions](#viewing-the-failure-trace-from-github-actions)
+* [Test Code Standards and Style Guide](#test-code-standards-and-style-guide)
+  * [Context Use](#context-use)
+  * [Sharing Code Between Steps](#sharing-code-between-steps)
+  * [Step wording](#step-wording)
+  * [Assertions](#assertions)
+  * [Step parameter types](#step-parameter-types)
+* [How the Tests Work](#how-the-tests-work)
+  * [Django Test Runner and Test Case](#django-test-runner-and-test-case)
+  * [Database Snapshot and Restore](#database-snapshot-and-restore)
+  * [Playwright](#playwright)
+* [Why Aren't We Using Existing Django Testing Modules?](#why-arent-we-using-existing-django-testing-modules)
+  * [Pytest-BDD](#pytest-bdd)
+  * [Behave-Django](#behave-django)
+<!-- TOC -->
+
 ## Structure
 
 Feature files live in the [features](./features) directory.
@@ -75,6 +99,23 @@ Then restore it with
 poetry run dslr --url postgresql://ons:ons@localhost:15432/ons restore <SNAPSHOT_NAME>  # pragma: allowlist secret
 ```
 
+## Running the Tests
+
+See the [main README functional tests section](/README.md#run-the-functional-tests) for the basic commands for running
+the tests.
+
+### Playwright Options
+
+Some Playwright configuration options can be passed in through environment variables
+
+| Variable   | Description                                                                                                                                                                    | Default                        |
+|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|
+| HEADLESS   | Toggle headless browser mode, set to "False" to show the browser window                                                                                                        | True                           |
+| SLOW_MO    | Sets the Playwright slow mo mode in milliseconds                                                                                                                               | 0                              |
+| BROWSER    | Set the browser for playwright to use, must be one of `chromium`, `firefox`, or `webkit`.<br/> NOTE: Currently only chromium is supported and tests may fail in other browsers | chromium                       |
+| TRACE      | Toggle Playwright trace recording                                                                                                                                              | True                           |
+| TRACES_DIR | Sets the location to write Playwright trace files if TRACE is enabled                                                                                                          | <working_directory>/tmp_traces |
+
 ## Viewing Failure Traces
 
 The tests record traces of all their actions, allowing you to follow through tests that previously ran and debug issues
@@ -97,37 +138,6 @@ Our GitHub Action is configured to save traces of any failed scenario and upload
 
 You should then be able to step through the failed tests and get a better idea of the state and cause of the failure.
 
-## How the Tests Work
-
-### Django Test Runner and Test Case
-
-Due to [issues with the Django TransactionTestCase](/TODO) which prevent us using the built-in database teardown/setup
-in between scenarios, we have implemented our own database snapshot and restore pattern between tests. We still make use
-of the Django test case, specifically
-the [LiveServerTestCase](https://docs.djangoproject.com/en/stable/topics/testing/tools/#django.test.LiveServerTestCase)
-to perform the initial database setup and run a live server on a random port for the tests.
-
-### Database Snapshot and Restore
-
-We are using [DSLR](https://pypi.org/project/dslr/) for fast database snapshots and restores.
-
-After we have used the Django test runner to set up the test database and initialise it by running migrations, we take a
-DSLR snapshot of this clean, initial state. In the test case fixture, post test, we then restore the clean snapshot,
-ensuring each test gets a clean, migrated database, isolated from other tests.
-
-This setup is done in behave fixtures, kept in [behave_fixtures.py](behave_fixtures.py). These fixtures are the
-registered in the behave hooks in the [environment.py](environment.py)
-
-### Playwright
-
-To give the scenario steps access to a playwright page, we set up a Playwright instance along with a browser and
-browser context in the `before_all` hook, so that it is started once at the beginning of the run. In the
-`before_scenario` hook, we then create a Playwright page object to be used by the scenario, passed to through the behave
-context. This page is closed in the `after_scenario` hook, to ensure each scenario has its own separate page object.
-
-If the `PLAYWRIGHT_TRACE` environment variable is enabled, we also start trace recording at the beginning of the run,
-and start a new trace "chunk" for each scenario, so that traces of individual failed scenarios can be saved to files.
-
 ## Test Code Standards and Style Guide
 
 ### Context Use
@@ -178,3 +188,66 @@ Where we need [step parameters](https://behave.readthedocs.io/en/stable/tutorial
 complex data than single strings or the other basic types supported by the default parser, we
 use [custom registered types](https://behave.readthedocs.io/en/stable/api.html#behave.register_type). These are
 registered in the [environment.py](environment.py) so they are available to all steps.
+
+## How the Tests Work
+
+### Django Test Runner and Test Case
+
+Due to issues with the Django TransactionTestCase which prevent us using the built-in database teardown/setup
+in between scenarios, we have implemented our own database snapshot and restore pattern between tests. We still make use
+of the Django test case, specifically
+the [LiveServerTestCase](https://docs.djangoproject.com/en/stable/topics/testing/tools/#django.test.LiveServerTestCase)
+to perform the initial database setup and run a live server on a random port for the tests.
+
+### Database Snapshot and Restore
+
+We are using [DSLR](https://pypi.org/project/dslr/) for fast database snapshots and restores.
+
+After we have used the Django test runner to set up the test database and initialise it by running migrations, we take a
+DSLR snapshot of this clean, initial state. In the test case fixture, post test, we then restore the clean snapshot,
+ensuring each test gets a clean, migrated database, isolated from other tests.
+
+This setup is done in behave fixtures, kept in [behave_fixtures.py](behave_fixtures.py). These fixtures are the
+registered in the behave hooks in the [environment.py](environment.py)
+
+### Playwright
+
+To give the scenario steps access to a playwright page, we set up a Playwright instance along with a browser and
+browser context in the `before_all` hook, so that it is started once at the beginning of the run. In the
+`before_scenario` hook, we then create a Playwright page object to be used by the scenario, passed to through the behave
+context. This page is closed in the `after_scenario` hook, to ensure each scenario has its own separate page object.
+
+If the `PLAYWRIGHT_TRACE` environment variable is enabled, we also start trace recording at the beginning of the run,
+and start a new trace "chunk" for each scenario, so that traces of individual failed scenarios can be saved to files.
+
+## Why Aren't We Using Existing Django Testing Modules?
+
+At first glance it may appear our custom fixtures and database restore mechanism should be unnecessary, as there are
+multiple choices for modules out there which claim to do what we need. We tried these solutions first and ruled them out
+because of various incompatibilities with our app or testing requirements.
+
+### Pytest-BDD
+
+This is built on Pytest, and we decided to move our unit and integration testing away from Pytest because it has
+compatibility issues with our multi-DB configuration, so this would have suffered the same issue.
+
+Live server testing is accomplished with a fixture, which under the hood uses a Django `LiveServerTestCase`. This
+inherits from
+the [TransactionTestCase](https://docs.djangoproject.com/en/stable/topics/testing/tools/#django.test.TransactionTestCase),
+which causes us serious compatibility issues, as it uses an isolated test database and flushes all data in between
+tests. We have migrations which seed critical data rows, so a flush operation breaks the app.
+
+We tried various workarounds such as using a fixture file to restore the data, but this runs into Wagtail issues, and
+even if it worked it would be non-ideal as that fixture file would have to be kept up to date and recreated when any new
+seeded data is added.
+
+### Behave-Django
+
+Behave-Django is a module which enables easier integration between the Behave BDD framework and a Django app. It uses
+the Django `StaticLiveServerTestCase` and Django test runner to wrap the Behave test runs. This means we run into the
+exact same data flushing issue as we did with Pytest-BDD.
+
+Also, the `StaticLiveServerTestCase` is incompatible with Whitenoise, which we use to serve static content, so we would
+have to override the test case. This was possible, but the setting was only exposed through command line arguments, so
+it would make running the scenarios through an IDE with debugging features either impossible or require much more manual
+setup.
diff --git a/functional_tests/environment.py b/functional_tests/environment.py
@@ -24,8 +24,8 @@ def before_all(context: Context):
     use_fixture(django_test_runner, context=context)
 
     context.playwright: Playwright = sync_playwright().start()
-    context.playwright_trace = str_to_bool(os.getenv("PLAYWRIGHT_TRACE", "True"))
-    context.playwright_traces_dir = Path(os.getenv("PLAYWRIGHT_TRACES_DIR", str(Path.cwd().joinpath("tmp_traces"))))
+    context.playwright_trace = str_to_bool(os.getenv("TRACE", "True"))
+    context.playwright_traces_dir = Path(os.getenv("TRACES_DIR", str(Path.cwd().joinpath("tmp_traces"))))
 
     configure_and_launch_playwright_browser(context)
 
@@ -40,7 +40,7 @@ def configure_and_launch_playwright_browser(context: Context) -> None:
     browser_type = os.getenv("BROWSER", "chromium")
     headless = str_to_bool(os.getenv("HEADLESS", "True"))
     slow_mo = int(os.getenv("SLOW_MO", "0"))
-    default_browser_timeout = int(os.getenv("DEFAULT_BROWSER_TIMEOUT", "10_000"))
+    default_browser_timeout = int(os.getenv("DEFAULT_BROWSER_TIMEOUT", "5_000"))
 
     browser_kwargs = {
         "headless": headless,