[PROPOSAL] Simplify OpenSearch distribution branching strategy

[andrross]

What/Why

What are you proposing?

We should change the branching strategy used across the OpenSearch Project to track two releases instead of three. We should make this change when we do the next major version release (3.0) of the distribution.

This change applies to all repositories that are part of the OpenSearch distribution (core, dashboards, plugins, etc). Some repositories, such as clients, have independent release cycles and are not impacted by this change.

Background

Currently, we track three releases in parallel:

• 3.0 (the next major release) on the main branch
• 2.x (the next minor release) on the 2.x branch
• 1.3.x (the next maintenance release) on the 1.3 branch

This strategy optimizes for being able to incubate and develop breaking changes for the next major release for an extended period of time, as that code would be on the main development branch. The cost of this strategy is that all non-breaking changes intended for the next minor release require an extra step of backporting to the 2.x branch. The previous 2.5+ years have shown us that we have incubated very few breaking changes on the main branch for an extended period. While we have automation that makes backports to the 2.x branch very low effort for the majority of changes, it is not perfect (e.g. automation hits conflicts requiring manual resolution, flaky tests may fail, it takes time even when everything works perfectly). I believe the cost of these paper cuts, multiplied by thousands of changes across scores of repos, outweigh the benefits of the current approach. I also have no reason to believe things would be different for the 3.x→4.0 cycle than it was for the 2.x→3.0 cycle.

Proposal

The proposal is to track only two releases in parallel. After the 3.0 release, we will have the following branches:

• main tracking the 3.x version (the next minor release)
• 2.<last>.x tracking the next maintenance release of the 2.<last> version

At the time of a minor release, we will cut a release branch (e.g. 3.1) from the main branch. Development for the following minor release will continue on main and the 3.1 version will be built from the 3.1 branch.

At the time of a maintenance release, we will cut a release branch (e.g. 2.19.1) from the 2.19.x branch. The maintenance release will then be built from the 2.19.1 branch.

What users have asked for this feature?

This is a change to development and will not impact users. Developers frequently complain about friction introduced by backports.

What problems are you trying to solve?

This proposal is an attempt to reduce development friction by optimizing for what we routinely do (develop new features in a backward compatible way targeted toward the next minor release) at the expense of what we do not do (incubate breaking changes for an extended period in a snapshot build of the next major release).

What is the developer experience going to be?

The vast majority of code changes are intended to be released in the next minor version. With this proposal, such changes will require only one PR to commit the change to the main branch. This is in contrast to the procedure today, which requires committing the change to main and triggering a backport to the 2.x branch.

Are there any security considerations?

No.

Are there any breaking changes to the API

No.

Why should it be built? Any reason not to?

We should do this to improve the developer experience across dozens of repos.

The downside is that we will have no dedicated place to stage future 4.0 changes or routinely build 4.0 snapshot distributions. If someone wants to incubate a breaking change they would need to create a separate branch and regularly rebase it against main. We would have no automated mechanism to build a complete 4.0 distribution containing breaking changes in multiple components.

What will it take to execute?

This is a change in branching convention that will have to be coordinated across all repos that are part of the distribution. Some release tooling in opensearch-build may need to be updated to handle the new convention.

[dblock]

This is a great proposal, as the amount of backport PRs we did in 2.x is huge.

Will we continue releasing patch releases often such as 3.1.1 after 3.1 before 3.2? Feels like we want that to be reserved to urgent high severity security issues and just do 3.1, 3.2, 3.3, etc. And we won't need a 3.x branch.

[andrross]

Will we continue releasing patch releases often such as 3.1.1 after 3.1 before 3.2? Feels like we want that to be reserved to urgent high severity security issues and just do 3.1, 3.2, 3.3, etc.

Patches of the current minor version (e.g. 3.1.1) only happen for high severity issues. If a 3.1.1 release is necessary we would cherry pick the critical fixes on to the 3.1 branch and then do a patch release from that branch. This proposal does not change that policy.

And we won't need a 3.x branch.

That's correct. With this proposal we track two releases which means only two active branches: main and 2.19.x. At the time of any release (either regularly scheduled or a patch) we cut a release branch from the appropriate branch, but we don't continue to maintain release branches after the release.

[msfroh]

I have a very minor preference for renaming main to 3.x because it's the branch that we cut 3.x releases from (i.e. 3.1, 3.2, etc.)

At the end of the day, it wouldn't change anything functionally. It would just feel a little closer to "Developing directly on 2.x", which of course is something we decided not to do during the 2.x line (but we're advocating for here).

As I said -- it's a very minor preference. If we decide to keep the name of the branch as main, I still think that avoiding a backport for every single PR (and the main -> 2.x BWC dance) is a great idea.

[peterzhuamazon]

I am on the same page with Andrew's proposal.

Overall it should simplify the process not only on code developer side, but also on infra and automation as we do not need to maintain a specific next version build that breaks very often due to instability of the code.

Thanks.

[dblock]

I have a very minor preference for renaming main to 3.x because it's the branch that we cut 3.x releases from (i.e. 3.1, 3.2, etc.)

This would be very surprising to anyone coming to the project and would make the main branch a moving target for dozens of repos.

[msfroh]

This would be very surprising to anyone coming to the project and would make the main branch a moving target for dozens of repos.

I don't understand the second clause. I'm suggesting that instead of having a branch named main, it would literally be called 3.x -- nothing moving. Since other repos (mostly?) take a dependency on built artifacts (which are versioned) rather than branches, I don't understand how it would impact them one way or another.

[andrross]

@msfroh I think @dblock is saying it would be surprising that the default branch would be a moving target, because it would change with every major version. The convention is that the name of the default branch is fixed (main or mainline etc) and always represents development on the latest version. @dblock please correct me if I'm wrong.

[dblock]

@andrross thanks, you said exactly what I was thinking

[peternied]

@msfroh I think there is value in making it obvious which version number is associated with main and I think it breaks 'standard' convention to remap HEAD to a version'ed branch instead of main as @andrross points out.

Cheap alternative, what would you think of a small piece of automation that kept a 3.x branch in lockstep with main on ever push?

---

@andrross Thanks starting this discussion - I'm in full support of creating a 3.0.0 release.

In conjunction with this major release revisiting the release methodology and messaging might be a good exercise, as once upon a time we did -alpha releases. At the very least do a pass over the release process and confirm if we are making any changes. I'd love to see a blog post 😄

[msfroh]

Cheap alternative, what would you think of a small piece of automation that kept a 3.x branch in lockstep with main on ever push?

I don't think that would be of much benefit to anyone. If I cared enough, I could create a branch named 3.x in my local repository setup to track the main branch from the project whenever I git pull. (Or I could name my local branch whatever I want, like zorgoth_destroyer_of_worlds.) In practice, I'm happy to go along with the consensus of sticking to the main name.