IDE-friendly unit tests

[proway2]

Right now tests created for page objects aren't discoverable by pytest, therefore IDEs (e.g. VS Code) aren't aware of tests presence, though there might be other unit tests discoverable by IDE. This results in the next problems:

• Even if you need to run tests just for a single netloc you've got to run either all tests or to specify a path to a given test, which is not very convinient if you work with few different netlocs at a time.
• web-poet implicitly registers a pytest plugin which, sometimes, is not needed. Disabling is possible, but you've got to use something like -p no:web-poet. Which is again makes the whole process less natural for the IDE.
• No standard way to run a single test for a single URL.
• Overall, the whole approach to unit tests varies drastically from the usual way pytest tests are created and used.

Would be really nice to have IDE-friendly tests for page objects that are:

• discoverable by pytest and IDE without the need to configure some low-level stuff.
• possible to run and debug in IDE a single test for a single URL.
• pytest plugin could be easily disabled if needed.

[kmike]

Hey! The problem clearly exists (you're having issues with VSCode and web-poet fixtures). I wanted to clarify some points though.

Right now tests created for page objects aren't discoverable by pytest

The tests are discoverable, as can be seen when running pytest using CLI. We're using official pytest API to write plugins: https://docs.pytest.org/en/7.1.x/how-to/writing_plugins.html, or see e.g. an example: https://docs.pytest.org/en/7.1.x/example/nonpython.html#yaml-plugin. So, the issue is not that tests are not discoverable, but that the VSCode doesn't seem to support all pytest features.

Even if you need to run tests just for a single netloc you've got to run either all tests or to specify a path to a given test, which is not very convinient if you work with few different netlocs at a time.

To run tests for a single netloc locally, one can pass path to the fixture to pytest. I think something like pytest my_proj/fixtures/mywebsite should work. I'm not using IDEs personally to run the tests (running them from the command line), but maybe it's possible to configure the IDE to support that - though that may not be possible, or it may depend on IDE.

web-poet implicitly registers a pytest plugin which, sometimes, is not needed. Disabling is possible, but you've got to use something like -p no:web-poet. Which is again makes the whole process less natural for the IDE.

Hm, why do you need to disable the plugin? Could you please elaborate; does it cause any issues?

No standard way to run a single test for a single URL.

Yeah, it's #144

Overall, the whole approach to unit tests varies drastically from the usual way pytest tests are created and used.

Right, but it's a standard and supported approach, documented in pytest.

It's not very common, but there are some useful packages which use this approach, e.g. https://github.com/davidfritzsche/pytest-mypy-testing (which is awesome, and which we use actively in several projects), or even https://github.com/pytest-dev/pytest-cpp. You may also argue that doctests don't follow the standard approach as well; they use custom discovery. There are also plugins like https://github.com/thisch/pytest-sphinx which help to keep examples in the docs up to date, by checking them.

The advantage of using .json fixtures is that they can be edited as text, and manipulated programmatically. We were planning e.g. to add some UI to generate and edit the fixtures, so that even less technical people can do it, and then pass it to the developers. It won't be possible if the tests are stored completely in some generated .py files.

If we want to preserve the advantages of .json fixture, then it's possible to have a hybrid approach: the data is still stored in .json files, but there are .py files with regular pytest tests, which load the test from the fixtures. The downside is duplication: now, instead of just data, we also have .py files (probably, auto-generated), which need to be kept in sync, and which can't (or shouldn't) be edited programmatically. For the basic use case - running the whole testing suite, or running a subset of tests in CLI, it doesn't seem to give any benefit, it's just more files, which user needs to keep in sync. But it could make it easier for IDEs to run these tests, so it may be a workaround.

[proway2]

but that the VSCode doesn't seem to support all pytest features

It's not about VSCode or any other IDE, it's about how fixtures are organized, these don't follow this doc

Hm, why do you need to disable the plugin? Could you please elaborate; does it cause any issues?

I've found a way to make tests compatible with IDEs, so there's no need for the plugin. It's safe to keep it registered, but it's about running all tests twice.

[kmike]

It's not about VSCode or any other IDE, it's about how fixtures are organized, these don't follow this doc

Sorry, I don't agree :) There are test types which don't follow this doc, e.g. all the doctest; they follow a different convention (https://docs.pytest.org/en/7.1.x/how-to/doctest.html). The conventions which you linked to are conventions for Python tests discovery specifically.

Pytest provides standard hooks to customize test discovery and test collection (see e.g. https://docs.pytest.org/en/7.1.x/reference/reference.html#collection-hooks), and uses it itself.

[proway2]

Either way the current approach doesn't play nice with IDEs.