Merge pull request onflow#6177 from onflow/alex/branching-conventions

iteration of branching convention

DEVELOPMENT.md

@@ -1,24 +1,117 @@
 # Development
 
-## Feature Release Flow
-
-### Branching Strategy
+Please carefully read our [Coding Conventions](./CodingConventions.md).
+
+## Branching Policy
+
+**This convention is _binding_**<sup>⭒</sup> **for all PR requests in this repo**, as
+deviations pose a significant risk to mainnet liveness. If you believe the branching convention could be improved, 
+please prepare a concrete proposal and add an agenda item for
+a [Core Protocol Working Group](https://github.com/onflow/Flow-Working-Groups/tree/main/core_protocol_working_group) meeting.
+
+(⭒) There may be scenarios where the Branching Policy has its limitations, and one-off deviations could be beneficial.
+This includes accidental deviations that, once discovered, are not easily undone.
+These cases should be carefully tracked and require awareness and detailed coordination.
+It is the responsibility of team members working on branches that break the policy to ensure
+the _entire_ engineering team working on `flow-go` is informed and stays in the loop.
+
+### Overview
+Conceptually, we maintain four categories of branches:
+* **deployment branches** contain software versions that were/are/should be running on testnet and mainnet.
+  The naming convention is `v0.minor` and loosely follows semantic versioning. The major version is currently pinned to `0`,
+  as the Flow protocol is not yet fully implemented. The minor version increases at every Height-Coordinated-Upgrade [HCU] or spork (points of breaking downwards compatibility).
+  In a nutshell, you can think of the minor as a version number for the overall protocol. Each time there is a change in the protocol, such that nodes
+  running the old and new version can't be mixed, the minor version increases and we create a new deployment branch
+  (usually from `master`).
+* `master` accumulates all revisions and features that are either fully downwards compatible or can be rolled out via an HCU. Probably the majority of PRs
+  can be merged directly to master or via a feature branch. Furthermore, our ability to roll out upgrades to the live network via HCUs will significantly increase over time.     
+* Most **feature branches** implement protocol changes small enough to be deployed via an HCU (and hence can be merged to `master`). 
+  However, for the foreseeable future, large and broadly-breaking improvements will be necessary for a fast evolution and
+  feature-completion of Flow. Such features should be kept in their individual feature branches.
+* The `spork` branch will be created in preparation of a new spork from the most recent `master`. Feature branches holding HCU-**in**compatible upgrades will be
+  consolidated on this `spork` branch. The spork branch is intended to be very short-lived and only exist for a few weeks right before the testnet spork.
+  When the Flow network has committed to the spork on a governance level (currently coordinated by the Flow Foundation), 
+  the `spork` branch will be merged back to `master`.
+
+We are purposefully continuing the usage of the 'spork' terminology here, to describe a "severely breaking change".
+Specifically, spork describes a deployment of new node software, which requires the entire network to be stopped and rebooted.
+We only carry over a snapshot of the execution state but fully re-initialize the protocol state. Thereby, major upgrades of the protocol,
+incl. migrations of the entire execution state to a new formats become reasonably straight forward.
+Additional context on HCU vs sporks can be found [here](https://developers.flow.com/networks/node-ops/node-operation/hcu#hcu-versus-spork).   
+
+
+<img src='./docs/flow-go_branching-convention.png' width='350'>
+
+
+
+
+#### Motivations
+* Based on the decentralized nature of blockchain networks, continuous deployment is practically intractable (without compromising decentralization).
+Therefore, longer wait times for features until their deployment must be accounted for in the development process. This convention has been incrementally
+developed and refined based on the multi-year learnings of managing the deployment of the flow network.
+* Deployment branches, `master` containing a runnable snapshot of the most recent development state and feature branches are intuitively exactly what you
+  would expect without further flow-specific context. We just _extend_ the common convention by a few edge cases accounting for the upgrade-constraints of a blockchain network:
+  * On a specific deployment branch, there can only be non-breaking changes. This is to prevent accidents where we roll out new node
+    software incrementally to one node after another, but then discover later that there is some specific case where the two versions don't work together and network halts. 
+  * Due to the limitations of upgrades that can be rolled out via HCUs, we have to separate spork-level features. That is the scenario we address with long-living
+    feature branches and the spork branch.   
+* You may wonder why we don't just use a single spork branch right away. So `master` would contain all HCU-compatible upgrades and the spork-branch _in addition_ also the HCU-**in**compatible upgrades.
+  We optimize for the case where the majority of changes is HCU-compatible, while we have very few HCU-incompatible features in the pipeline which in tendency go very deep in one specific area of the code base.
+  If we collected the HCU-incompatible features in a single 'spork' branch, we would need to merge master into that. That means an engineer would need to know all HCU-incompatible features
+  currently in the pipeline to resolve merge conflicts. In our model where teams develop features in parallel, it has proven to be efficient for each team to maintain their own HCU-incompatible feature
+  branches and regularly merge the evolving `master` into them. As feature branches stay aligned with `master`, the last remaining step of merging all the feature branches together is usually not a big lift.      
+* During the limited lifetime of the `spork` branch, the evolution of `master` should be very small. Generally, the engineering team is more focused on testing, a significant portion of features will have
+  completed their development in time for the spork and new features start in their own branches anyway.      
+
+
+### The [master](https://github.com/onflow/flow-go/tree/master) branch
+
+The `master` branch is intended to only contain features for the _immediately_ upcoming release.
+This is under the assumption that we already committed to the type of upgrade (HCU vs Spork).
+
+**Generally only HCU-compatible changes are allowed to be merged to `master`**. It is the responsibilities of the developers to provide conclusive evidence why
+their change can be deployed via HCU. For fully downwards-compatible changes, this explanation can be omitted. Otherwise, please provide a brief summary in your PR
+targeting master as of why the code changes are HCU-compatible (for example "only affects transaction execution and verification"). More details are provided in the 
+[breaking change classification](#breaking-change-classifications) section below.
+
+
+### Deployment branches
+
+- For every HCU and spork, a new deployment branch will be created from master. This branch will be tagged and used to update testnet and then mainnet.
+- **Only non-breaking changes** can be committed to a deployment branch, such that nodes running the older and newer version on this branch can be mixed.
+  In other words, a single deployment branch spans _all_ patches that are protocol-compatible without an HCU. 
+- Each time there is a breaking-change in the protocol (see [breaking change classification](#breaking-change-classifications) below),
+  the minor version increases and we create a new deployment branch from `master`.
+
+#### Naming Convention:
+The naming convention is `v0.minor`, with the major version currently pinned to `0`.
+The minor version increments at every Height-Coordinated-Upgrade [HCU] or spork (points of breaking downwards compatibility).
+An example is the branch `v0.33`. Extensions of the branch names are allowed but optional, for example including key differences
+such as cryptographic stack (e.g. `v0.33-relic`), storage implementation (e.g. `v0.33-storehouse`), etc.  Patch versions (e.g. `0.33.0`) are usually only included in tags and builds. 
+
+
+### Feature branches
+- During development, all features should live on a feature branch.
+- For small features, this will be a simple working branch. These branches have the naming scheme `<contributor>/<issue #>-<short description>`, for example `kan/123-fix-known-issue`
+- For larger features, this may be a shared feature branch with other team members. Feature branches have the naming scheme `feature/<name>`.
 
-#### The [master](https://github.com/onflow/flow-go/tree/master) branch
+### The `spork` branch
 
-- The `master` branch is intended to only contain features for the immediately upcoming release, whether it is a [Height Coordinated Upgrade (HCU) or a Spork](https://developers.flow.com/networks/node-ops/node-operation/hcu#hcu-versus-spork).
+The spork branch was specifically introduced for managing spork-level changes. Sporks are very far apart (targeted at 12 months intervals), so the `spork` branch is not needed for the time in between sporks.
+HCU-compatible features are merged to `master` and HCU-incompatible features live in their own feature branches until shortly before the spork.
+In an ideal world, where we finalize the spork date a few months ahead of time and _always_ stick to it, the `spork` branch would not be needed at all. 
 
-#### HCU or Spork specific branches
+Despite our best efforts, occasionally circumstances arise that require us to postpone the spork just a few weeks before it was supposed to happen.
+In fact, given the magnitude and depth of changes being rolled out in a spork, needing to postpone with short notice is actually a reasonably plausible scenario.
+And the `spork` branch helps us to handle this scenario: 
 
-- For every HCU and spork, a branch will be created from master. This branch will be tagged and used to update testnet and then mainnet.
-- Only non-breaking changes (see: [breaking change classification](#breaking-change-classifications)) can be committed to these HCU or spork specific branches.
+Unless we are absolutely sure the spork is going to happen when planned, limit `master` to HCU-compatible changes only. Thereby we keep our options open
+to postpone the spork and if needed to continue with HCUs (e.g. for time-sensitive security fixes). However, we also need to consolidate our
+HCU-**in**compatible (major breaking) changes, resolve conflicts and test to keep our options open for actually sporking on the planned date,
+which we do on the `spork` branch. 
 
-#### Feature branches
-- During development, all features should live on a feature branch.
-- For small features, this will be a simple working branch. These branches have the naming scheme `<contributor>/<issue #>-<short description>, for example `kan/123-fix-known-issue`
-- For larger features, this may be a shared feature branch with other team members. Feature branches have the naming scheme `feature/<name>`.
 
-### Upgrade Path Eligibility
+## Upgrade Path Eligibility
 
 - When a feature branch is ready to be merged, the desired upgrade path onto Mainnet must be determined (if any). The options are:
     - Height Coordinated Upgrade (HCU)
@@ -35,17 +128,19 @@
 - All HCU upgrades can go directly into the `master` branch
 - All spork upgrades must live on their own feature branch until the last HCU before the spork has been performed (usually approximately 1 month before the Spork).
     - It is the responsibility of the DRI to keep this feature branch in a mergeable state.
-    - If the spork is scheduled to occur within a month, all the feature branches can be merged into `master`. However, if the exact spork date has not been decided then a special spork branch may be created from master to merge all the feature branches. This is to consolidate all the feature branches while accommodating any additional HCUs that may occur between then and the spork date.
+    - If the spork is scheduled to occur within a month, all the feature branches can be merged into `master`.
+      However, if the exact spork date has not been decided, then a special `spork` branch may be created from master to merge all the feature branches.
+      This is to consolidate all the feature branches while accommodating any additional HCUs that may occur between then and the spork date.
     - Suggestion: once a sprint, merge `master` into the feature branch. More frequent merges are easier, as they avoid complex conflict resolutions
 
 
-### End of Release Cycle
+## End of Release Cycle
 
 - At the end of every release cycle, we will tag a commit that includes all desired features to be released
 - This commit will be tagged according to [semantic versioning guidelines](https://dapperlabs.notion.site/Changes-to-handling-git-tags-5e39af7c723a428a915bd88901fc1274)
 - Release notes must be written up, describing all features included in this tag
 
-### Benchmark Testing
+## Benchmark Testing
 
 [Benchmarking](https://www.notion.so/Benchmarking-e3d89e3aadb44b0787da9bb7703b0dae?pvs=21)