Deduplicate code between Hydra and Nix

[Ericson2314]

Currently Hydra duplicates lots of functionality that is in Nix. This is bad because it leads to either the Nix or Hydra versions of a feature being undertested and broken (e.g. remote builds are very finicky), or missing features on one side (Hydra doesn't support ssh-ng://).

The solution is to dedup, fixing both problems, and lowing our maintenance burden across the board.

I will try to break this down into steps, updating this list accordingly

Easier

• Generic SSH logic

  • Nix: src/libstore/ssh.cc in Nix
  • Hydra: top of src/hydra-queue-runner/build-remote.cc

  Notable differences: Hydra passes a number of extra flags to ssh, evidently based on sysadmin war stories.

• Legacy SSH protocol

  • Nix: src/libstore/legacy-ssh-store.cc in Nix
  • Hydra: Much of src/hydra-queue-runner/build-remote.cc

  Notable differences: Hydra doesn't issue builds blocking and synchronously, so it can cancel them and handle logs as they occur. Probably other things, TBD.

  • Split up logic so subsequent refactors are easier to both write and review Split the buildRemote function #1180

  • ServeProto::Command::QueryValidPath

    • Nix: Factoring out parts of the serve protocol for Hydra to share nix#6134
    • Hydra: Begin factoring out the protocol code #1165

  • ServeProto::Command::ImportPaths

  • ServeProto::Command::BuildDerivation

    • Hydra part-way: Use factored-out BuildResult serializer #1318

  • ServeProto::Command::QueryPathInfos

    • Nix: Factor out ServeProto::Serialiser<UnkeyedValidPathInfo> and test nix#9560

  • ServeProto::Command::DumpStorePath

• Use Store abstraction after previous two are complete. Support arbitrary stores for building #1200

• Use the same notion of a Machine config for remote builders. NixOS/nix@ebc9f36 was a Nix-side refactor to prepare for this, but then the Hydra side was not done.

  • Use Nix's Machine type in a minimal way #1341 Use Nix's Machine type
  • Dedup with nix: use nix::Machine::parseConfig #1386 Use Nix's Machine parser

  See also Why is machine loading rounded? #484

  See also Triage and break up the global settings nix#5638, where we would like more of the machine stuff to go in the Store config. That ties this to the previous checkbox.

Farther off

• Duplicate Scheduling logic, Nix's build/*.cc vs Hydra

CC @rickynils @Ma27 @grahamc

[Ericson2314]

Note to fix this, it would be very convenient to use a local build of Nix during Hydra development. I will try to hack that up.

edit it is indeed possible!

[Ericson2314]

https://discourse.nixos.org/t/developing-a-system-that-replaces-nix-remote-build/22388/11 lists a huge number of issues with the current remote building situation that makes using Nix in large institutional contexts difficult. CC @elaforge

The immediate goals of this issue are to shrink the amount of the code an increase testing coverage by moving us towards a world where hydra and remote building both battle-test the same code paths. But a further goal would be to, be improving the quality and flexibility of the build scheduling code, make experimentation with the alternative architectures suggested in that that thread easier.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/developing-a-system-that-replaces-nix-remote-build/22388/13

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-02-nix-team-meeting-minutes-13/23731/1

[fricklerhandwerk]

Related: #1200

• @thufschmitt: agreed with the idea, the maintenance overhead of duplicated code is significant
  • @edolstra: the big duplication is that Hydra has its own build loop, which
    • @thufschmitt: why does that exist in the first place?
    • @edolstra: for optimisation. later it was not obvious how to port it back to Nix
• @edolstra: related: Support arbitrary stores for building #1200
• @thufschmitt: key question: is anyone willing to spend significant time on this? sounds like months of work
  • @fricklerhandwerk: this would be a worthwhile to have funding for, because it unblocks lots of further development, such as RFC92 (dynamic derivations) and stabilising RFC62 (content-addressed derivations), separate store types
    • @Ericson2314: we also need resources for reviews, otherwise the work will never land
      • right now it ultimately revolves around trust. we could leverage different processes, such as delegating more involved reviews to pairs of team members