Suggestion: drop Go 1.17 when Go 1.18 is released

[marten-seemann]

I'd like to suggest to make an exception to our backwards-compatibility guarantee of always supporting the most recent two Go versions. This exception would only cover the transition from Go 1.17 to Go 1.18, when Go 1.19 is released, we'd go back to supporting the two most recent versions (1.18 and 1.19).

why?

Go 1.18 will add support for generics, which is arguably the biggest (and most anticipated!) language change. I expect that people will want to use these new features as soon as possible. At least I myself can't wait to get my hands dirty!

There are two use cases we need to distinguish here:

1. non-API changes: These will be possible even if we keep the backwards-compatibility guarantee, by using build flags. Ultimately, you'd end up with two copies of the respective code, one that does and one that doesn't use generics.
2. API changes: For some packages, it might be desirable to expose generic types, i.e. interfaces, types and functions that have a type parameter.

I see both of these cases as problematic, to the point of effectively prohibiting the use of generics:

1. Writing generic code, while at the same time keeping a non-generic version around practically duplicates large pieces of code, doubling the maintenance work in case we find a bug in that piece of code (note that the bug might exist in the non-generic or the generic version of the code, or both).
2. Exposing an API that uses generics will be impossible if we still want to support Go 1.17 (unless you're willing to expose different interfaces for Go 1.17 and 1.18, which would make the API very hard to use).

how?

We probably want to start testing using Go 1.18 as soon as the RC is released, which (at the least) means using binaries built using the RC on our own infrastructure. This is not a particularly dangerous thing to do, after all, Google is confident enough in the performance / correctness of Go RCs that they use it for their entire infrastructure. Should that reveal any potential problems with Go 1.18, we could hold off on this plan and decide Go 1.17 for a bit longer, until those problems are fixed.

cc @mvdan @guseggert @aschmahmann @Stebalien @vyzo @BigLep @lidel @raulk

[mvdan]

What about making this opt-in? One could opt in by adding go 1.18 or later to their go.mod. Most of our modules right now have go 1.16, and can stick with go 1.17 if they don't want to add generics just yet.

I disagree with dropping support for 1.17 across all our repos once 1.18 is out - for the vast majority of our software, breaking 1.17 as soon as 1.18 is out won't be a good idea. Upgrading to the latest stable version of Go is easy for you and me, but it's a process that can take weeks or months once projects get larger or have their own release schedules. We need those six months.

[mvdan]

I should also note that generics won't simply be finished in 1.18. It's very likely that 1.19 and later releases will further tweak generics or add more features, such as golang/go#48918. So I don't think a sudden break with 1.17 will solve all of our problems - we'll be at practically the same situation once 1.19 comes out.

You're right that 1.18 will be a very big release, though I think that by default we should still treat it like a normal release in terms of backwards compatibility and support.

[marten-seemann]

What about making this opt-in? One could opt in by adding go 1.18 or later to their go.mod. Most of our modules right now have go 1.16, and can stick with go 1.17 if they don't want to add generics just yet.

I would agree, if we weren't building a software stack. If only a single module starts using generics (without doing the backwards-compatibility dance, which I think is impractical), go-ipfs and lotus won't build with Go 1.17 any more. So leaving this decision to maintainers is not really an option, we need to make an org-wide decision: Do we want to (practically, see my post above) forbid the use generics for another half a year, for the sake of keeping compatibility with 1.17, or do we want to be able to use the new features?

I should also note that generics won't simply be finished in 1.18. It's very likely that 1.19 and later releases will further tweak generics or add more features, such as golang/go#48918. So I don't think a sudden break with 1.17 will solve all of our problems - we'll be at practically the same situation once 1.19 comes out.

That's annoying (and at the same time, a completely understandable decision from the Go team). I can't really foresee how big the difference between 1.18 and 1.19 is going to be. Maybe we need to extend the exception to 1.19 as well, if the diff turns out to be equally significant?

[mvdan]

we need to make an org-wide decision

For libraries, I fully agree. I should have clarified that I think opting into dropping 1.17 should only be a possibility for non-libraries, such as end user programs or deployed services.

The only other path I see is that we say our web3 stack only supports one Go version. Personally, I think that would be a bad idea for the reasons I outlined above. A good example is how go-ipfs generally lags behind by one major Go version, or about six months - due to its release schedule, but also how each upgrade ends up causing some work due to the large amount of dependencies. I imagine Lotus is in a similar spot.

So, to me, supporting only one Go version also implies forcing those projects to upgrade Go versions much faster. I'm not sure how easy that would be; it feels like it's not worth the tradeoff against waiting six more months to use generics in production libraries.

[mvdan]

Maybe we need to extend the exception to 1.19 as well, if the diff turns out to be equally significant?

If the proposal is to only support one Go version for the next 2-3 releases, then my opinion is definitely that we should not do that :)

[mvdan]

All the above said, if the stewards and team leads agree that generics is important enough that we want to support only one Go version temporarily across the stack, I'm happy to help implement that. My intuition is that it will add an amount of pain and work that won't be worth skipping the six-month wait, though it's also largely not my decision.

My only point of view is go-ipld-prime; it would certainly be nice to use generics in some of the core APIs, but I also don't think it's reason enough to break downstream users sooner than necessary. We simply have bigger fish to fry in the short term, too.

[marten-seemann]

A good example is how go-ipfs generally lags behind by one major Go version, or about six months - due to its release schedule, but also how each upgrade ends up causing some work due to the large amount of dependencies. I imagine Lotus is in a similar spot.

I don't think the release schedule is the problem here. We might need to do some extra work to upgrade the Go version (we need to do this anyway, as we're on Go 1.16 now, just that we'd upgrade to Go 1.18). I'm convinced the benefits far outweigh the hassle this causes.
If Go 1.18 is released in Feb, and the next go-ipfs is only happening in April (say), that wouldn't pose any problems. Switching a package to Go 1.18 would require a major release (or minor for pre-v1 packages), so we'd even be able to cut patch releases, if that becomes necessary.

So, to me, supporting only one Go version also implies forcing those projects to upgrade Go versions much faster. I'm not sure how easy that would be; it feels like it's not worth the tradeoff against waiting six more months to use generics in production libraries.

I evaluate that tradeoff completely differently: It only takes a single maintainer (within PL) upgrading their repo to 1.18 to force us all to do follow along. Also, it only takes a single 3rd party library cutting a release that drops 1.17 support to do the same. Given the appeal of generics, it will almost be inevitable that this will happen sooner or later.
It will be really hard telling people that they can't use generics for another half a year.

I do realize that we've been telling people so far that they can't use new language features for half a year, but this release is different. There's a difference between being able to use net.ErrClosed and a language feature that's unblocking use cases and APIs that people have been literally begging for for the last 10 years.

Just for myself, here's two examples:

• I'd love to get rid of a bunch of code-generated code in quic-go. I already have a patch that adds generics support, and the code is so much cleaner! I'm hesitant to merge it though, since it's not a trivial change (see here and here), and keeping two implementations of the same struct in parallel will just be a nightmare if there's a bug in that code.
• Earlier today, I wrote up a proposal how to do a long-due refactor of go-multistream. If not allowed to use generics, one would have to add enormous amounts of boilerplate code (or use code generation), AND break the API twice (once for the non-generic API change, then again for the change that's allowed to use generics). If this is the tradeoff, this would probably need to remain unfixed for the next 9 months. This is an excellent example for how our "2 most recent releases" policy would cause us to hold back a change that's necessary and beneficial for the project.

I'm sure that we'll find a lot more examples like these, as soon as Go 1.18 is released and people start playing with generics.

[guseggert]

Re: adopting generics in our core libraries, my preference is to take a similar approach as the Go team is taking with the standard library--wait for the dust to settle and for folks to figure out all the tradeoffs, pitfalls, best practices, etc., so we don't end up with API designs that we later regret. It's hard for me to be confident in any predictions about long-term maintenance costs of code using / designed around Go generics, since nobody in the world has any experience with that.

AFAIU, The purpose of the "2 most recent releases" policy is to give consumers time to upgrade their Go versions. So a major factor here should be "how much cumulative consumer pain will there be?", and then "is this an acceptable amount of pain for the payoff?". It's not clear to me from skimming those examples what the concrete alternatives look like, their relative user payoffs and maintenance costs, and how hard they would be for consumers to upgrade to.

Earlier today, I wrote up a proposal how to do a long-due refactor of go-multistream. If not allowed to use generics, one would have to add enormous amounts of boilerplate code (or use code generation), AND break the API twice (once for the non-generic API change, then again for the change that's allowed to use generics).

Another option here is to deal w/ the boilerplate/codegen, and not break the API later.

If only a single module starts using generics (without doing the backwards-compatibility dance, which I think is impractical), go-ipfs and lotus won't build with Go 1.17 any more.

IIUC, it's more nuanced than that, it depends on if the generic code survives tree shaking and is actually compiled. So in some cases it may not be as bad as it sounds.

So leaving this decision to maintainers is not really an option, we need to make an org-wide decision: Do we want to (practically, see my post above) forbid the use generics for another half a year, for the sake of keeping compatibility with 1.17, or do we want to be able to use the new features?

I disagree that we need to make a sweeping decision about this right now. Library authors should be free to make a decision about that, provided they make a reasonable case about the tradeoffs. But it should be specific and they should build consensus with the library's major consumers (direct and transient, as it affects both).

Also relevant is that there were a number of dependencies that broke from 1.16->1.17 and we were glad to have the slack to work through those over time, without it becoming a huge disruption for us and consumers.

[marten-seemann]

Re: adopting generics in our core libraries, my preference is to take a similar approach as the Go team is taking with the standard library--wait for the dust to settle and for folks to figure out all the tradeoffs, pitfalls, best practices, etc., so we don't end up with API designs that we later regret.

I'm not sure if this is the right tradeoff for us. The Go team is bound by the Go 1 compatibility promise, so they have to double- and triple-check every API choice they make. We're not bound by such a rigorous policy, semver and go modules allow us to make backwards-incompatible changes when necessary (and we do it literally all the time). I'd rather gain experience by using the new features than watch from the sideline.

I disagree that we need to make a sweeping decision about this right now. Library authors should be free to make a decision about that, provided they make a reasonable case about the tradeoffs.

Every author of every library that we use would have to make the same conservative tradeoff as we do. For this Go update, the opportunity cost for keeping backwards compatibility is huge, and we might be forced to drop 1.17 support due to a random 3rd party package we depend on anyway. I'd rather not pay that cost and start reaping the benefits of generics right away.

Maybe there's a middle ground here: We could decide to keep Go 1.17 for a certain time after Go 1.18 has been released (one month? until 1.18.1 is released?), to give consumers (and ourselves) some more time to do the necessary updates.

[ribasushi]

Every author of every library that we use would have to make the same conservative tradeoff as we do.

This is not an accurate argument. In addition to the above, stewards would need to change go.sum to be affected. Historically this is done extremely rarely and conservatively within the main (💸-affecting) offerings, roughly once a year. Thus the argument "we would be affected by what authors of our libraries do" does not follow.

It would be extremely concerning if lotus, go-ipfs and estuary stop building with 1.17 before 1.18 is at a minimum 6 months old.

[marten-seemann]

Every author of every library that we use would have to make the same conservative tradeoff as we do.

This is not an accurate argument. In addition to the above, stewards would need to change go.sum to be affected. Historically this is done extremely rarely and conservatively within the main (💸-affecting) offerings, roughly once a year.

As soon as any library starts using generics, and we need to pull in an updated version (e.g. for an unrelated bug fix in that library, or because it adds another feature that we want), we wouldn't be able to use Go 1.17 any more.
Not sure I understand your comment about go.sum, That file is auto-generated from go.mod when running go mod tidy, I assume you mean go.mod? Bumping the version number in go.mod doesn't prevent the package from building with older Go versions though, the only thing that prevents building with older Go versions is the use of features that were added in newer versions.

[vyzo]

I think we should wait for the dust to settle before generifying, we have a lot of downstream dependents.

[anorth]

If only a single module starts using generics (without doing the backwards-compatibility dance, which I think is impractical), go-ipfs and lotus won't build with Go 1.17 any more.

This sounds like the kind of thing that versioning is for. Certainly if a library introduced generics in its API then that's a major version change, so projects depending on the old API should be fine until they explicitly upgrade. I would suggest that requiring a new compiler even internally would similarly warrant this, given Go's source-based build system.

If a library wants to serve its downstream consumers well (what other purpose does it have?) it should maintain the version/branch that the majority of downstream use until only a small minority do. This might mean backporting some fixes, etc. It's fine for new feature development to happen only in the v2 branch, but then downstream gets to opt in to it and can decide about upgrading compilers at that point. That is, I think it's fine if "I want the new stuff" forces downstream projects to upgrade compiler so they can get it, but not "I just want it to keep working".

This might finally be the thing that pushes the non-versioned (i.e. v0 or v1) libraries into being versioned properly.

we need to make an org-wide decision

We're not just an org, but an ecosystem. Making an org-wide decision sounds fundamentally counter to a productive open-source-style collaboration. We cannot make the decision for all our library consumers. If we find ourselves in a situation where we need to make an org-wide decision, IMO we need to change something else about our development practises so that it's no longer the case, and we support heterogeneity.

[edit]

we might be forced to drop 1.17 support due to a random 3rd party package we depend on anyway

If some upstream library screws us in this way (i.e. without versioning), we should fork it until we're ready to make an opt-in upgrade.

[hannahhoward]

FWIW, I like the @anorth proposal -- if you're using generics, bump a major version and let people pick. Support and update the old version at least till we get to go 1.19 or a majority of downstream users switch, and then you can say the old version is frozen (actually I have no idea what our general support guarantees are for old major versions of libraries... that seems like an important question also to figure out as we transition away from "org wide" and move to "ecosystem wide")

[vyzo]

I dont understand this rush to generify -- libp2p is not a playground!

[mvdan]

I think the consensus is to not do this, and Go 1.19 is now only four months away, given how late 1.18 was. So I imagine we want to simply wait for 1.19 to drop 1.17 as usual.

[marten-seemann]

Agreed. Let's close this.