tests/nixpkgs: move nixpkgs module test to dedicated drv

[MattSturgeon]

We don't need a full test that actually builds or runs nixvim. All we need is some assertions on the result of a nixvim configuration.

Currently #2738 needs to change the module tests to construct their own pkgs instance, so that we can test how the module behaves. However this is likely to have a performance hit since we are repeatedly instanciating pkgs again for each test case. It's probably better to use a single pkgs instance for all test case modules, but move the nixpkgs-module tests to their own dedicated test derivation.

This PR does that, moving the existing tests to a dedicated link-farm.

This also allows creating some bespoke helpers to streamline writing the tests and may improve performance slightly because we don't need to also build config.build.package for the nixpkgs-module unit tests.

The bespoke helpers are a little cumbersome, so I'm open to feedback on them. Some of them may also be useful more generally, so may be worth moving to some kinda "test utils" file. This can be done now or later.

[MattSturgeon]

FYI #2738 is kinda blocked on this. Or at least blocked on deciding whether we will merge this.

I'd like to re-write the tests in that PR based on this PR.

[traxys]

I'd be in favour of introducing in this PR the framework for tests that don't launch nvim, it can be useful to add tests for assertions or warnings (#2751 reminded me that I wanted to add a diagnostic & such a test)

[MattSturgeon]

I'd be in favour of introducing in this PR the framework for tests that don't launch nvim, it can be useful to add tests for assertions or warnings (#2751 reminded me that I wanted to add a diagnostic & such a test)

That's still on my radar, but I don't see how it relates to this PR

[traxys]

Those tests don't require to launch nvim, so I guess what I am mostly saying is that it would be nice to create in this PR a test suite that does not eval nvim, and allows to easily add tests to it

[MattSturgeon]

Ok, now that we've tackled negative tests can we revisit this PR and discuss whether we will merge it or something like it?

While we could set test.buildNixvim = false and keep the nixpkgs module tests in test-sources, that'd require all the test-spurces modules to not define nixpkgs.pkgs.

IDK how performance intensive instantiating an instance of nixpkgs is, or how well nix is able to cache similar instances for re-use...

Given the sheer number of test case modules we have, I feel it's probably a good idea to explicitly use the same pkgs instance for all of them.

If so, that means keeping the nixpkgs module tests separate, as per this PR (or similar).

[traxys]

Yeah it addresses my comment, so I have no idea about the performance though :/

Stale

[MattSturgeon]

I've re-worked this PR with a much nicer implementation.

It still achieves the same separation, but is much cleaner by using config.build.test instead of writing runCommand tests from scratch.

[traxys]

Indeed, much cleaner!

[MattSturgeon]

@mergify queue

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 5b068e7

[mergify]

This pull request, with head sha 5b068e7f8f2b6beaa1fafe0c8b3604b63bcccc2d, has been successfully merged with fast-forward by Mergify.

This pull request will be automatically closed by GitHub.

As soon as GitHub detects that the sha 5b068e7f8f2b6beaa1fafe0c8b3604b63bcccc2d is part of the main branch, it will mark this pull request as merged.

It is possible for this pull request to remain open if this detection does not happen, this usually happens when a force-push is done on this branch nixpkgs_test, this means GitHub will fail to detect the merge.

[GaetanLepage]

Thanks @MattSturgeon !