Skip to content

Branches and releases

Jump to bottom
John Kerl edited this page Feb 20, 2024 · 60 revisions

Things everyone should know

  • Everyone should commit to main as a general rule
  • Micros are tagged on branches such as release-1.5, release-1.6, release-1.7
    • The first will be 1.5.0, 1.6.0, 1.7.0, etc
    • If there are subsequent PRs on a branch we can make a 1.5.1, 1.6.1, etc.
  • When you merge a PR to main, simply add a tag backport release-1.5, backport release-1.6, etc. This will put up a PR for that branch.
  • Timing:
    • Recall major.minor.micro: the minor is the second number and the micro is the third.
    • I know of no scenario which would cause core's major to bump, nor TileDB-SOMA's.
    • Whenever core bumps a minor, dependent packages including TileDB-SOMA bump a minor.
      • Example: core 2.19 appears, TileDB-Py 0.25 is tagged depending on 2.19, TileDB-R 0.23 is tagged depending on 2.19, then a TileDB-SOMA PR can be put up (on main) depending on 2.19, 0.25, and 0.23
      • When core 2.20 appears (along with TileDB-Py 0.26 and TileDB-R 0.24) we will tag a TileDB-SOMA 1.8 -- regardless of how much or how little development work has happened in the TileDB-SOMA repo since 1.7.
    • If it so happens that there is a significant new feature in TileDB-SOMA, we reserve the right to bump the TileDB-SOMA minor even if there hasn't been a core release yet. As of January 2024, this hasn't happened yet, but it could.
    • When core bumps a micro -- e.g. 2.19.0 to 2.19.1 -- we may or may not bump our micro.
      • If the core change doesn't affect TileDB-SOMA at all, we might have no need to bump our micro.

Things the release engineer should know

Roles

  • TileDB-SOMA is an open-source collaboration between TileDB and CZI
  • Most of this checklist, except the Conda and TileDB-Cloud steps, can be done by someone at either company
  • Someone at TileDB needs to be involved to complete the Conda and TileDB-Cloud steps

Deciding

  • Post in the common Slack channel that you want to do a release
    • Include a list of the PRs that will be in the release — the GitHub UI lets you auto-generate this list
    • Wait for two people to agree
  • Validate the version number and wait for two people to agree
    • We use semantic versioning which is seemingly unambiguous: majors for backward-incompatible changes, minors for new features that are backward-compatible, micros for bugfixes
    • But in reality this is subjective: for example, if something was a bit incomplete before, was its incompleteness a bug being fixed, or is the new bit a new feature?
    • Be sure to also bump the tiledbsoma version at apis/r/DESCRIPTION, as this lives outside of GitHub versioning
  • Considering the scope of changes in the release, a release candidate (e.g. i.j.krc0, with no - before rc0, for PEP386) might be required if:
    • If new dependencies, end-to-end validation is required
    • If new features are introduced, validation of features is required
    • If there are removals of APIs, validation of downstream software is required
    • If there are deprecations, proper release notes with explanations and replacements outlines is required
    • If a release candidate was created, all validations must be completed and successful to cut the release

Validating

  • CI in the TileDB-SOMA repo is our #1 signal
  • Conda CI (below)
  • Notebook evaluations
    • At first, manually evaluate them
    • Over time: automate notebook-evaluations in a CI job

Branch/tag mechanics

  • RELEASE BRANCH: For each minor (i.j++.0) make a branch release-i.j but don’t make a tag yet
    • If we need to backport bugfixes from main later, we’ll be glad we had this
    • Also we’ll need this just below
    • The moment you make a new release branch like release-1.8, also make a new label like backport release-1.8 -- this will allow the backport bot (discussed above) to work
  • R VERSION: Before publishing a particular Git tag somewhere on the branch, put up a PR on the this repo bumping the package version in apis/r/DESCRTIPTION and apis/r/NEWS.md, as the R package gets its version not from the GitHub tag
  • FEEDSTOCK PRE-CHECK PR: Before publishing a particular Git tag somewhere on the branch, put up a PR on the feedstock repo with the SHA of the commit you would be tagging
  • GITHUB TAG: Publish a git tag and GitHub release, trigger CI to build, and publish PyPI packages.
    • Pro-tip: for i.j.0 releases, consider a i.j.0rc0 first (note: no - before rc0, for PEP386)
    • Include in the release notes all the PRs from the autogenerated changelog but organize them as follows:
      • New features added
      • Breaking changes
      • Minor changes
      • Bug fixes
      • Upcoming deprecation notices
    • See https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-notes for how to manually upload to PyPI if needed
  • R DEV VERSION: After a new release-i.j has been created, put up a PR on main setting apis/r/DESCRIPTION version to i.j.99: for example https://github.com/single-cell-data/TileDB-SOMA/pull/2005. Subsequent commits on main will be i.j.99.1, i.j.99.2, etc.
  • CELLXGENE-CENSUS UPDATE: Put up a PR on cellxgene_census to have it depend on the new TileDB-SOMA version
    • Without this, any Python user/system that installs both tiledbsoma and cellxgene_census will get the older version not the newer
  • FEEDSTOCK POST-TAG PR: Conda and R (i.e. CRAN) will require manual updating/triggering of updating pipeline.
    • Another feedstock PR but with the final SHA. And this one to actually merge, not just put up as a CI canary like the previous one was
    • Same recipe/meta.yaml
    • Beware rc vs main release channel as here: https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/46/files.
    • Pro-tip, this tiledbsoma-feedstock CI takes about half an hour to run
    • But update sha and url this time, with git_url and git_rev commented back out
    • The sha256 is computed from TileDB-SOMA-i.j.k.tar.gz which can be found at https://github.com/single-cell-data/TileDB-SOMA/releases/tag/i.j.k
    • Re-render locally. This will do a git commit on your behalf in your local repo: git push this.
conda install -n root -c conda-forge conda-smithy
# or perhaps
mamba install conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

# or keeping your base env clean
mamba create --yes -n rerender conda-smithy
conda activate conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

If

RuntimeError: conda-smithy version (3.24.1) is out-of-date (3.26.3) in conda-smithy's environment. Exiting.

then

mamba upgrade -c conda-forge conda-smithy -n base

and retry the above.

Artifacts

Note that all Conda artifacts for dependent packages must be present in order to Conda-ify TileDB-SOMA.

Aside for arm64 (M1/M2) Macs

  • As noted on https://github.com/single-cell-data/TileDB-SOMA/pull/1385, using GitHub Actions as of mid-2023 the best we can do for MacOS arm64 is to cross-compile for arm64 while actually running on x86_64.
  • This is tracked here: https://github.com/github/roadmap/issues/528 -- TL;DR maybe 2023Q4 GHA will have this available
  • This also means we can build the arm64 wheels in CI but we cannot smoke-test them -- since we do not have arm64 hardware to run the smoke test on
    • Consolation prize:
    • Get the artifacts from the CI job
    • Download and unzip wheels-macos-arm64.zip
    • pip install tiledbsoma-i.j.k-cp311-cp311-macosx_11_0_arm64.whl (or whatever your OS version)
    • In Python, import tiledbsoma and tiledbsoma.show_package_versions()
    • Run unit tests via python -m pytest apis/python/tests

Bug-fix branches

Problem to solve:

  • TileDB-SOMA 2.n exists, depending on core 2.m et al.
  • We need to bug-fix TileDB-SOMA 2.(n-1), which depends on core 2.(m-1) et al.
  • The R DESCRIPTION-file syntax does not support upper bounds, so any codemod pushed to a non-latest branch will result in the PR on TileDB-SOMA 2.(n-1) getting a version of TileDB-R for core 2.m, unless we explicitly force it not to.
  • Discussion: https://github.com/single-cell-data/TileDB-SOMA/pull/1993#issuecomment-1881150316
  • Solution examples:
    • #1972 pins TileDB-R to 0.21 for the release-1.5 branch
    • #1996 pins TileDB-R to 0.22 for the release-1.6 branch

Using TileDB-SOMA

Developing TileDB-SOMA

Clone this wiki locally