xen: Add test-ring0 tests

[dinhngtu]

Run the test-ring0 tests that are straightforward to run.
The ones that are not (e.g. legacy tests, tests that crash the host or leak resources infinitely) are skipped, but included for completeness.

test_xst_memory_leak requires a custom kernel with CONFIG_DEBUG_KMEMLEAK set.

[ydirson]

Why do we want to ignore modprobe's exit status here (and on similar cases)?

[dinhngtu]

Mostly since failure to remove test modules has no bearing on the test results, and the host is cleaned up afterwards by rebooting anyway.

[ydirson]

Shouldn't that rather be an error?

[ydirson]

I feel it would be easier to read as something like

if host.ssh(...):
    pytest.error(...)

[ydirson]

Without prior context, it is hard to understand what we're testing here. A detailed description and/or more comments seem to be needed

[stormi]

Needs to be reordered. Straight imports first, alphabetically. Then a blank line. Then "from xxx" imports, alphabetically.

[stormi]

As Yann indicated above, I forgot about the split between external and internal libs, too.

[ydirson]

from typing here does not bother me much, and pep8 indeed does not seem to mandate any sorting within each of the 3 import blocks. IMO we should check what various checkers report.

[stormi]

Stack Overflow most voted answer pushes for sorting, and that's also the convention we tend to use.

https://stackoverflow.com/questions/20762662/whats-the-correct-way-to-sort-python-import-x-and-from-x-import-y-statement

[ydirson]

I thought you were originally refering to pytest not being sorted 😉

[stormi]

We avoid skipping tests for such reasons. One might falsely think that the test was executed, in CI, as no one checks all skips one by one. What we do instead is check for prerequisites in a fixture and fail the test if the prerequisite is not met.

[ydirson]

I would not use fail, that's rather what error is for

[stormi]

If that's appropriate, then we have a few fixtures to fix.

[dinhngtu]

I couldn't find pytest.error anywhere. I'll use pytest.fail.

[stormi]

By moving the prerequisite test to a fixture as I suggested (and that's one of the things fixtures are here for in pytest), it would be obvious that the failure is in the prerequisites and not in the test itself.

[ydirson]

Yes, my bad, there does not seem to be another mechanism than fixtures to generate error rather than failure (https://docs.pytest.org/en/stable/explanation/fixtures.html#fixture-errors seems to be where they talk about it, but it does not spell out precisely that errors are solely coming from fixtures)

[ydirson]

Next question would be, do we want to do this in a host_without_livepatch_tester fixture to be able to get an error rather than a failure?

[stormi]

That's precisely what I'm talking about from the beginning :D (though in other fixtures we do use pytest.fail too at the moment)

[stormi]

As it is, the test will be executed as part of the xen job. This job is currently very short and runs at the beginning of our test pipeline, on real hardware. However, with a reboot, we'll go from 1 minute to close to 10 minutes.

Are the tests themselves long to execute?

Our options:

• keep as is
• move to a separate job that we'll run later in the pipeline
• split between fast tests that would not require a reboot (if any) and the rest.

[dinhngtu]

The tests themselves are quite short. I can split it to a separate job if desired.

[stormi]

We can keep them in the same job for now.

[stormi]

Please update the xen job description in jobs.py to indicate that it will reboot the master host.