|
| 1 | +--- |
| 2 | +feature: Defined conventions around testing of official Nixpkgs packages. |
| 3 | +start-date: 2021-12-29 |
| 4 | +author: Jonathan Ringer |
| 5 | +co-authors: |
| 6 | +shepherd-team: @Mic92 @Artturin @kevincox |
| 7 | +shepherd-leader: @Mic92 |
| 8 | +related-issues: |
| 9 | + - [RFC 0088 - Nixpkgs Breaking Change Policy](https://github.com/NixOS/rfcs/pull/88) |
| 10 | +--- |
| 11 | + |
| 12 | +# Summary |
| 13 | +[summary]: #summary |
| 14 | + |
| 15 | +When updating or modifying packages, several conventions for testing regressions |
| 16 | +have been adopted. However, these practices are not standard, and generally it's not well |
| 17 | +defined how each testing method should be implemented. It would be beneficial to have |
| 18 | +an unambiguous way to say that a given package, and all downstream dependencies, have |
| 19 | +had as many automated test ran possible. This will give a high degree of certainty that |
| 20 | +a given change is less likely to manifest regressions once introduced on a release |
| 21 | +channel. |
| 22 | + |
| 23 | +Another desire of this rfc is also to have a way for various review tools |
| 24 | +(e.g. ofborg, hydra, nixpkgs-review) to have a standard way to determine if a |
| 25 | +package has additional tests which can help verify its correctness. |
| 26 | + |
| 27 | +# Motivation |
| 28 | +[motivation]: #motivation |
| 29 | + |
| 30 | +Breakages are a constant painpoint for nixpkgs. It is a very poor user experience to |
| 31 | +have a configuration broken because one or more packages fail to build. Often when |
| 32 | +these breakages occur, it is because the change had a large impact on the entirety |
| 33 | +of nixpkgs; and unless there's a dedicated hydra jobset for the pull request, it's |
| 34 | +infeasible to expect pull request authors to verify every package affected |
| 35 | +by a change they are proposing. However, it is feasible to specify packages that |
| 36 | +are very likely to be affected by changes in another package, and use this information |
| 37 | +to help mitigate regressions from appearing in release channels. |
| 38 | + |
| 39 | +# Detailed design |
| 40 | +[design]: #detailed-design |
| 41 | + |
| 42 | +Standardize `tests.<name>` (commonly set with the `passthru.tests.<name>` argument to `mkDerivation`) as a mechanism of |
| 43 | +more expensive but automatic testing for nixpkgs. As well as encourage the usage of |
| 44 | +`checkPhase` or `installCheckPhase` when packaging within nixpkgs. |
| 45 | + |
| 46 | +Criteria for `tests.<name>`: |
| 47 | +- Running tests which include downstream dependencies. |
| 48 | + - This avoids cyclic dependency issues for test suites. |
| 49 | +- Running lengthy or more resource expensive tests. |
| 50 | + - There should be a priority on making package builds as short as possible. |
| 51 | + - This reduces the amount of compute required for everyone reviewing, building, or iterating on packages. |
| 52 | +- Referencing downstream dependencies which are most likely to experience regressions. |
| 53 | + - Most applicable to [RFC 0088 - Nixpkgs Breaking Change Policy](https://github.com/NixOS/rfcs/pull/88), |
| 54 | +as this will help define what breakages a pull request author should take ownership. |
| 55 | +- Running integration tests (E.g. nixosTests) |
| 56 | + - Tests which have heavy usage or platform requirements should add the appropriate systemFeature |
| 57 | + - E.g. `nixos-test` `kvm` `big-parallel` |
| 58 | + |
| 59 | +Usage for mkDerivation's `checkPhase`: |
| 60 | +- Quick "cheap" tests, which run units tests and maybe some addtional scenarios. |
| 61 | +- Since this contributes to the "build time" of a package, there should be some |
| 62 | +emphasis on ensuring this phase isn't bloated. |
| 63 | + |
| 64 | +Usage for mkDerivations `installCheckPhase`: |
| 65 | +- A quick trivial example (e.g. `<command> --help`) to demonstrate that one (or more) |
| 66 | +of the programs were linked correctly. |
| 67 | +- Assert behavior post installation (e.g. python's native extensions only get installed |
| 68 | +and are not present in a build directory) |
| 69 | + |
| 70 | +# Drawbacks |
| 71 | +[drawbacks]: #drawbacks |
| 72 | + |
| 73 | +None? This is opt-in behavior for package maintainers. |
| 74 | + |
| 75 | +# Alternatives |
| 76 | +[alternatives]: #alternatives |
| 77 | + |
| 78 | +Continue to use current ad-hoc conventions. |
| 79 | + |
| 80 | +# Unresolved questions |
| 81 | +[unresolved]: #unresolved-questions |
| 82 | + |
| 83 | +How far should testing go? |
| 84 | +- What consistitutes that "enough testing" was done to a package before a change was merged? |
| 85 | + |
| 86 | +Hydra: How would this look for hydra adoption and hydraChecks? |
| 87 | + |
| 88 | +# Future work |
| 89 | +[future]: #future-work |
| 90 | + |
| 91 | +One problem with onboarding more tests to the current nixpkgs CI and processes is the increased |
| 92 | +need of compute, storage, and ram resources. Therefore, consideration of future work should |
| 93 | +take into consideration how much testing is feasible for a given change. |
| 94 | + |
| 95 | +Onboarding of CI tools to support testing paradigms: |
| 96 | +- nixpkgs-review |
| 97 | + - Run `<package>.tests` on affected packages |
| 98 | + - Allow for filtering based upon requiredSystemFeatures |
| 99 | +- ofborg |
| 100 | + - Testing of `<package>.tests` is already done. |
| 101 | + |
| 102 | +Nixpkgs: |
| 103 | +- Add existing nixosTests to related packages |
| 104 | +- Update testing clause on PR template |
| 105 | +- Update contributing documentation |
0 commit comments