Move to golang modules instead of git submodules

[roidelapluie]

I would really love to see git modules go away in favour of golang modules.

I am trying to build mgmt inside $customer infra and will have to maintain an internal fork of mgmt with modules to be able to build it.

Go modules enable more reproducible builds, and they are pushed by golang upstream, we should use them.

[purpleidea]

@roidelapluie Is the only problem you're having the lack of reproducibility or is there anything else?

I was planning to solve the reproducibility issue, and I was likely going to use git modules for it, however the plan was to add those mod files into a subdir such as go.mod. and a make restore and make addtag script. Because I wanted git master to stay free of those, but still make it possible to reproducibly build tagged releases.

If you'd like to write such a script, that's welcome.

[roidelapluie]

Our pipelines internally use go mod, so that would be the easiest for us. Also normally we do go mod vendor and commit vendor/ in our internal repo.

Submodules would mean we need to mirror X repos internally, not that easy.

[jlucktay]

Hi there, speaking as someone that only just today cloned mgmt and started hacking through the quick start guide, FWIW my 2c is that going through that process without a go.mod file in the project root was my single biggest source of frustration in getting a binary built.

The maturity of Go modules has improved since this issue was raised; at the time Go 1.13 had only just come out, and then it wasn't until 1.14 dropped in Feb 2020 that they actively encouraged users to migrate.

Keeping vendor/ sub-directories as submodules is a pattern I hadn't come across before.
Switching to Go modules would definitely enable reproducible builds of git tags, if that is your main concern @purpleidea. 🙂

[purpleidea]

@jlucktay Can you describe specifically, in technical terms "my single biggest source of frustration in getting a binary built." ?
We also have binary releases which you can download and use directly.
Thanks!

[jlucktay]

It centred around the make-deps.sh script.
When it calls go get -v -t -d ./... it would use either 1) a version of an external package already downloaded in the local $GOPATH, or 2) pull whatever the latest version is from online, both of which are potentially incorrect/inconsistent, rather than their versions/commit hashes being explicitly defined/compatible.

I couldn't find any git submodule maintenance commands defined in the Makefile or the helper scripts either, so beyond the one initial git clone --recursive call listed in the quick start guide, there didn't seem to be any way to bring the mgmt workspace back to a known good state in terms of those vendor sub-directories.

I can take a swing at a PR with a Go module initialised, if it would be of any help?

[purpleidea]

@jlucktay

When it calls go get -v -t -d ./... it would use

So the goal for git master is to be git master. This keeps it stable every day for development, rather than having to all of a sudden on release day update 100 deps. There's an FAQ entry about this. Releases are a different story.

any way to bring the mgmt workspace back to a known good state in terms of those vendor sub-directories.

We'd like to do this for tagged release versions. This was discussed in a few places including in: #580 and #610

I can take a swing at a PR with a Go module initialised, if it would be of any help?

It would definitely help. The one caveat is that we want this wrapped in a shell script/Makefile so that the go.mod stuff is stored in a separate directory and not the root. This way, when I cut a release I can "build" one of these files (with your script) and if I want to use a specific one, then I can "restore" one of these (with your script).

If this makes sense, please let me know! We're happy to mentor the patch. This would let us get closer to reproducible builds, and also make it easier for you if you want to build that way.

[jlucktay]

After reading through that FAQ entry and the linked issues, the philosophy/approach taken here does make sense; use currently-available dependencies at build time and eat the breakages, weighed against ongoing maintenance.

If a go.mod file was in place, each commit to this repo would be snapshotting dependencies automagically.
As I understand it though, you'd like to keep a growing library of snapshots, go.mod.<timestamp> or go.mod.<commit hash> or some such.
I think that is doable with some Bash and the -modfile flag.

In starting to set it up however, I've hit an inconsistency with docker/docker (afraid it is the first of many):

mgmt/engine/resources/docker_container_test.go

Lines 72 to 73 in 76ede10

	Filters: filters.NewArgs(
	filters.KeyValuePair{Key: "id", Value: id},

In the filters package, NewArgs() doesn't currently take any arguments.

In v1.4.1 of docker/docker which is what has been pinned with the git submodule, the /api/types/filters package doesn't exist yet.

[purpleidea]

@jlucktay

the philosophy/approach taken here does make sense

Great!

If a go.mod file was in place, each commit to this repo would be snapshotting dependencies automagically.

I don't understand this comment. But my goal is to have git master have all git master deps other than what we must pin (because their git masters are broken) and for release tags to have an associated snapshot of all the git master deps from when I run git tag to cut a release.

As I understand it though, you'd like to keep a growing library of snapshots, go.mod. or go.mod. or some such.

Correct.

I think that is doable with some Bash and the -modfile flag.

Awesome, can you work on the patch please?

I've hit an inconsistency with docker/docker (afraid it is the first of many)

I don't know what's up here, but I'm confident they're easy to solve. Only the bash scripting and dealing with the golang tools sucks (because I'm not psyched about how the golang tooling works).

In v1.4.1 of docker/docker which is what has been pinned with the git submodule, the /api/types/filters package doesn't exist yet.

This might all be due to the random hack I did here:

https://github.com/purpleidea/docker/tree/445197336ebfc341fe1c922410324422b5722328

I can fix this up if it's blocking your patch from being merged once you post an initial version and I'll see how it's failing/etc

LMK what else you need!
Thanks!

[purpleidea]

WIP of trying to move to go mod is here:

https://github.com/purpleidea/mgmt/tree/feat/go-mod-without-consent

If someone who knows what they're doing wants to add some additional patches onto this, I'll squash it all and merge it.

/cc @roidelapluie @frebib

[zimbatm]

I'm not seeing any .gitmodules file in a fresh checkout. This issue is probably solved?

[frebib]

Yeah, Go modules is used for dependency management

[frebib]

@purpleidea This should be closed