Migrate to go modules: incompatible (>=v2) packages

[lanzafame]

Tracking issue for the migration of ipfs deps that are incompatible with go modules, i.e. they have a git tag that is semver and >=v2. This spans packages in both ipfs and libp2p.

Packages:

• github.com/ipfs/go-ipfs-cmds
• github.com/ipfs/go-datastore
• github.com/libp2p/go-libp2p
• github.com/libp2p/go-libp2p-circuit
• github.com/libp2p/go-libp2p-crypto
• github.com/libp2p/go-libp2p-host
• github.com/libp2p/go-libp2p-kad-dht
• github.com/libp2p/go-libp2p-kbucket
• github.com/libp2p/go-libp2p-metrics
• github.com/libp2p/go-libp2p-net
• github.com/libp2p/go-libp2p-peer
• github.com/libp2p/go-libp2p-peerstore
• github.com/libp2p/go-libp2p-pnet
• github.com/libp2p/go-libp2p-record
• github.com/libp2p/go-libp2p-routing
• github.com/libp2p/go-libp2p-secio
• github.com/libp2p/go-libp2p-swarm
• github.com/libp2p/go-stream-muxer

Along the way, I will attempt to migrate any incompatible packages that are dependencies of the above listed packages as well.

[Stebalien]

/cc

@ipfs/go-team, @libp2p/go-team

[anacrolix]

Can you define incompatible? It makes it sound like they can't work at all, when I think instead there may be a Go-mod specific meaning.

[lanzafame]

@anacrolix

i.e. they have a git tag that is semver and >=v2.

is the definition of incompatible.

[Kubuxu]

I think the consensus was to remove those tags and revert to v0 or v1.

[lanzafame]

@Kubuxu that is how we are going to progress forward at this point. I had wanted to maintain the existing git tags but it is more work than it is worth especially considering the inconsistency of gx releases with matching git tags.

[raulk]

@Kubuxu the latest idea was to prefix all existing tags with gx- and adapt gx release to produce gx- tags. That way the tag namespace is vacated for Go mod, but we preserve the gx lineage under a different namespace, as we’ll want to keep producing gx releases for now. This approach feels more like archiving history vs deleting history.

[Kubuxu]

I agree.

I'm all for not removing history (one of the reasons I created: https://github.com/ipfs/go-ipfs-archived/)

[bigs]

i'm into that @raulk @Kubuxu. could we also ship, with that update, a migration that checks to see if your local copy has been updated and gives you the option to do so if not? could be a git hook.

[anacrolix]

Not using the current major versions in libp2p was driven by incomplete third-party tooling support in libp2p/go-libp2p-kad-dht#236. As mentioned there, the core Go tooling works seamlessly with >= 2 major versions in import paths. Resetting versions will only aid support until version 2+, when again the issue with out-of-date tooling will pop up. On the downside, reverting to version 0 and 1 for projects may cause a lot of confusion, there is likely package managers other than gx in use that depend on the existing versioning, not to mention the human element of associating commits to version numbers. Many conversations around versioning in any project that has reset the version will begin with "version X? do you mean before/after the gx/version reset...". Many of these projects have been around for many years, I think pretending that they're still in v0 is vane and optimistic. Major versions are not a finite resource, and I think people are too easily encumbered by trying to make major versions a very serious thing.

[raulk]

@anacrolix gx doesn't use version numbers, it uses hashes; version numbers are purely vanity, that's why archiving them under a gx- tag namespace won't have implications for gx.

People using govendor and other third party tools should just upgrade to the official solution: gomod. We can't support every single ecosystem tool out there.

My point is that, prior to gomod, bumping up numbers in gx was somewhat done recklessly because it had no implications other than the text of the tag in Git.

However, with gomod, the implications of those version numbers are now visible. Not in metadata, but they actually surface in source files, yikes.

I strongly refuse making users have to go through this pain:

import (
	libp2p "github.com/libp2p/go-libp2p/v6"
	disc "github.com/libp2p/go-libp2p-discovery/v2"
	libp2pdht "github.com/libp2p/go-libp2p-kad-dht/v5"
	inet "github.com/libp2p/go-libp2p-net/v10"
	pstore "github.com/libp2p/go-libp2p-peerstore/v4"
	proto "github.com/libp2p/go-libp2p-protocol/v3"
	multiaddr "github.com/multiformats/go-multiaddr/v9"
	logging "github.com/whyrusleeping/go-logging/v800"
)

We are unlike most Go projects. We are an extremely modular project with independent version numbers that follow no predictable heuristic.

For the Kubernetes monorepo, using v4 (or whatever) all throughout is fine, because all imports will carry that number. We are not like that.

The above example is proof and it's horrendous DX. We'd be insane to accept inflicting that pain on our users, for no compelling gain.

To recap:

1. References to gx releases are hashes, not numbers. I don't expect ambiguity that cannot be resolved with a simple question.
2. gx vanity version numbers start with v1.0.0. Most modules are past v1. If we start gomod with v0.0.1, we have an entire [v0.0.1-v1.0.0) spectrum that's fully unambiguous.
3. To point 2, add the fact that most modules are beyond gx v1. For example, kad-dht is on gx v4.4.x, and if we trust our users to upgrade promptly, it's unlikely people are still running v3, so that makes the de facto unambiguous range [v0.0.1-v4.0.0).
4. Some devs on the team feel libp2p is pre-v1 in general, so this is a healthy reset. The alternative is straightjacketing us into a version lineage that had very different implications in the past (none), than it does now in the present with gomod (see above example).

[whyrusleeping]

@anacrolix's point about the confusion of resetting to <v1 versions is a good one. That sucks. But I think that what @raulk points out about the version numbers in the import paths sucks worse. This is being stuck between a rock and a hard place...

[anacrolix]

@anacrolix gx doesn't use version numbers, it uses hashes; version numbers are purely vanity, that's why archiving them under a gx- tag namespace won't have implications for gx.

It will for any other systems that rely on the tags.

People using govendor and other third party tools should just upgrade to the official solution: gomod. We can't support every single ecosystem tool out there.

Yes, although anybody that already has could have issues when the tags change under them.

However, with gomod, the implications of those version numbers are now visible. Not in metadata, but they actually surface in source files, yikes.

I strongly refuse making users have to go through this pain:

import (
	libp2p "github.com/libp2p/go-libp2p/v6"
	disc "github.com/libp2p/go-libp2p-discovery/v2"
	libp2pdht "github.com/libp2p/go-libp2p-kad-dht/v5"
	inet "github.com/libp2p/go-libp2p-net/v10"
	pstore "github.com/libp2p/go-libp2p-peerstore/v4"
	proto "github.com/libp2p/go-libp2p-protocol/v3"
	multiaddr "github.com/multiformats/go-multiaddr/v9"
	logging "github.com/whyrusleeping/go-logging/v800"
)

Note that this pain is only when you adopt go mod in "compatible" mode. Other packaging schemes are entirely unaffected with the right Go versions in use (for instance GOPATH users, and gx should be unaffected by major version use in paths, other systems may not). There are still replace directives and other tricks to make transitioning more gradual for go mod users. There's also the always ignored fact that you can still use go mod in the poorly-named "incompatible" mode, should you wish to avoid littering your import paths with major versions for a spell.

I think resetting the version is great if it eases the transition for consumers, but I'm not sure that it's necessary, or even correct to do it just because some linting tools break. Note that everything else in the downstream worked correctly without modification, even with major versioning throughout the import paths due to the core Go tool support.

I'm not hardline against the reset, but it smells of the aforementioned excitement that major version numbers always engender. If for example, we set go-libp2p-kad-dht to version 1 so the import paths don't change, as soon as some breaking changes are put in (for traversal stats and metrics for example), we go to version 2 and downstream linters break again. (And we definitely don't want our tools dictating changes.) If we went with version 0, we'd have downstream breakages fairly regularly due to the widespread dependence on the module and inability to communicate breaking changes.

• https://semver.org/#faq
• https://github.com/golang/go/wiki/Modules#faqs--semantic-import-versioning

[vyzo]

Let's not make a mess with the versions in the import paths.
Retagging the gx releases, as suggested by @raulk, is the correct path forward imho.