Skip to content

Branches and releases

Jump to bottom
John Kerl edited this page Sep 11, 2024 · 61 revisions

Things everyone should know

  • Everyone should commit to main as a general rule
  • Micros are tagged on branches such as release-1.5, release-1.6, release-1.7
    • The first will be 1.5.0, 1.6.0, 1.7.0, etc
    • If there are subsequent PRs on a branch we can make a 1.5.1, 1.6.1, etc.
  • When you merge a PR to main, simply add a tag backport release-1.5, backport release-1.6, etc. This will put up a PR for that branch.
  • Timing:
    • Recall major.minor.micro: the minor is the second number and the micro is the third.
    • I know of no scenario which would cause core's major to bump, nor TileDB-SOMA's.
    • Whenever core bumps a minor, dependent packages including TileDB-SOMA bump a minor.
      • Example: core 2.19 appears, TileDB-Py 0.25 is tagged depending on 2.19, TileDB-R 0.23 is tagged depending on 2.19, then a TileDB-SOMA PR can be put up (on main) depending on 2.19, 0.25, and 0.23
      • When core 2.20 appears (along with TileDB-Py 0.26 and TileDB-R 0.24) we will tag a TileDB-SOMA 1.8 -- regardless of how much or how little development work has happened in the TileDB-SOMA repo since 1.7.
    • If it so happens that there is a significant new feature in TileDB-SOMA, we reserve the right to bump the TileDB-SOMA minor even if there hasn't been a core release yet. As of January 2024, this hasn't happened yet, but it could.
    • When core bumps a micro -- e.g. 2.19.0 to 2.19.1 -- we may or may not bump our micro.
      • If the core change doesn't affect TileDB-SOMA at all, we might have no need to bump our micro.

Things the release engineer should know

Roles

  • TileDB-SOMA is an open-source collaboration between TileDB and CZI
  • Most of this checklist, except the Conda and TileDB-Cloud steps, can be done by someone at either company
  • Someone at TileDB needs to be involved to complete the Conda and TileDB-Cloud steps

Incrementing core versions

Example scenario:

  • Currently TileDB-SOMA depends on core 2.20, TileDB-Py 0.26, and TileDB-R 0.24.
  • The latest release branch is release-1.8.
  • Now a 2.21 has been released.
  • We need to:
    • Have main depend on 2.21, 0.27, and 0.25
    • Create a release-1.9 branch
    • Ensure that release-1.8 continues to get TileDB-R 0.24 (now that 0.25 is existing)

What needs to be done to make that happen:

  • For a PR on the main branch:
    • Run scripts/update_tiledb_version.py 2.21.0 (or whatever version)
    • Edit apis/python/setup.py to bump the TileDB-Py dependency to 0.27 (or whatever version)
    • Edit apis/r/DESCRIPTION to bump the TileDB-R dependency to 0.25 (or whatever version)
    • All that should go into a single PR like https://github.com/single-cell-data/TileDB-SOMA/pull/2150
  • For a PR on the latest release branch (release-1.8) in this example:
  • After all of this is stable in main, you can make a release-1.9 branch (or whatever version) following the procedure below in this document

What to wait for:

If we don't wait for all of those to be true -- if there's some urgency in getting the release out before all R binaries are available -- we need to do things like this:

and then this change will need to be undone in a separate PR a day or two later once all the above binary artifacts are available.

Deciding on a release

  • Post in the common Slack channel that you want to do a release
    • Include a list of the PRs that will be in the release — the GitHub UI lets you auto-generate this list
    • Wait for two people to agree
  • Validate the version number and wait for two people to agree
    • We use semantic versioning which is seemingly unambiguous: majors for backward-incompatible changes, minors for new features that are backward-compatible, micros for bugfixes
    • But in reality this is subjective: for example, if something was a bit incomplete before, was its incompleteness a bug being fixed, or is the new bit a new feature?
    • Be sure to also bump the tiledbsoma version at apis/r/DESCRIPTION, as this lives outside of GitHub versioning
  • Considering the scope of changes in the release, a release candidate (e.g. i.j.krc0, with no - before rc0, for PEP386) might be required if:
    • If new dependencies, end-to-end validation is required
    • If new features are introduced, validation of features is required
    • If there are removals of APIs, validation of downstream software is required
    • If there are deprecations, proper release notes with explanations and replacements outlines is required
    • If a release candidate was created, all validations must be completed and successful to cut the release

Validating

  • CI in the TileDB-SOMA repo is our #1 signal
  • Conda CI (below)
  • Notebook evaluations
    • At first, manually evaluate them
    • Over time: automate notebook-evaluations in a CI job

Branch/tag mechanics

  • RELEASE BRANCH: For each minor (i.j++.0) make a branch release-i.j but don’t make a tag yet
    • If we need to backport bugfixes from main later, we’ll be glad we had this
    • Also we’ll need this just below
    • The moment you make a new release branch like release-1.8, also make a new label like backport release-1.8 -- this will allow the backport bot (discussed above) to work
  • R MAIN-BRANCH VERSION: After a new release-i.j branch has been created, put up a PR on main setting apis/r/DESCRIPTION version to i.j.99: for example https://github.com/single-cell-data/TileDB-SOMA/pull/2005. Subsequent commits on main will be i.j.99.1, i.j.99.2, etc.
  • CZI ACCEPTANCE TESTS: As soon as there's a stable commit hash of main with all dependent packages updated -- but without waiting for the TileDB-SOMA tag -- loop in CZI for acceptance tests. They may find bugs needing fixing before we do the TileDB-SOMA tag.
    • Python census-reader acceptance
    • R census-reader acceptance
    • Python-only census-builder acceptance
  • SOMACORE FEEDSTOCK CHECK: It's possible that in our apis/python/setup.py and .pre-commit-config.yaml we've bumped a somacore version but not the associated Conda feedstock
  • FEEDSTOCK PRE-CHECK PR: Before publishing a particular Git tag somewhere on the branch, put up a PR on the feedstock repo with the SHA of the commit you would be tagging
  • R RELEASE-BRANCH VERSION: Immediately before publishing a particular Git tag somewhere on the branch, put up a PR on this repo bumping the package version in apis/r/DESCRTIPTION and apis/r/NEWS.md, as the R package gets its version not from the GitHub tag
  • PYPI STORAGE-SPACE CHECK:
  • GITHUB RC TAG: Same as next step, but, for i.j.0 releases, always do a i.j.0rc0 first (note: no - before rc0, for PEP386)
    • Too much can change in the dependency chain
    • It's possible for PyPI wheel-builds to fail immediately post-tag, requiring you to have to immediately turn around and do an i.j.1. Use the rc to avoid this.
  • WHEELS CHECK: Go to https://github.com/single-cell-data/TileDB-SOMA/actions and do a manual run of the "sdist & wheels" on the branch you're intending to tag -- unless the previous nightly was successful. If it wasn't, absolutely do this first before continuing, since if you don't, PyPI publish will fail later on. If you did an RC tag then you needn't do this, as the RC tag will run the wheels-build -- but do check GitHub Actions status for all-green.
  • GITHUB TAG: Publish a git tag and GitHub release, trigger CI to build, and publish PyPI packages.
  • CELLXGENE-CENSUS UPDATE: Put up a PR on cellxgene_census to have it depend on the new TileDB-SOMA version
    • Without this, any Python user/system that installs both tiledbsoma and cellxgene_census will get the older version not the newer
  • FEEDSTOCK POST-TAG PR: Conda and R (i.e. CRAN) will require manual updating/triggering of updating pipeline.
    • Another feedstock PR but with the final SHA. And this one to actually merge, not just put up as a CI canary like the previous one was
    • Same recipe/meta.yaml
    • Beware rc vs main release channel as here: https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/46/files.
    • Pro-tip, this tiledbsoma-feedstock CI takes about half an hour to run
    • But update sha and url this time, with git_url and git_rev commented back out
    • The sha256 is computed from TileDB-SOMA-i.j.k.tar.gz which can be found at https://github.com/single-cell-data/TileDB-SOMA/releases/tag/i.j.k
    • Re-render locally. This will do a git commit on your behalf in your local repo: git push this.
conda install -n root -c conda-forge conda-smithy
# or perhaps
mamba install conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

# or keeping your base env clean
mamba create --yes -n rerender conda-smithy
conda activate conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

If

RuntimeError: conda-smithy version (3.24.1) is out-of-date (3.26.3) in conda-smithy's environment. Exiting.

then

mamba upgrade -c conda-forge conda-smithy -n base

and retry the above.

Artifacts

Note that all Conda artifacts for dependent packages must be present in order to Conda-ify TileDB-SOMA.

Aside for arm64 (M1/M2) Macs

  • As noted on https://github.com/single-cell-data/TileDB-SOMA/pull/1385, using GitHub Actions as of mid-2023 the best we can do for MacOS arm64 is to cross-compile for arm64 while actually running on x86_64.
  • This is tracked here: https://github.com/github/roadmap/issues/528 -- TL;DR maybe 2023Q4 GHA will have this available
  • This also means we can build the arm64 wheels in CI but we cannot smoke-test them -- since we do not have arm64 hardware to run the smoke test on
    • Consolation prize:
    • Get the artifacts from the CI job
    • Download and unzip wheels-macos-arm64.zip
    • pip install tiledbsoma-i.j.k-cp311-cp311-macosx_11_0_arm64.whl (or whatever your OS version)
    • In Python, import tiledbsoma and tiledbsoma.show_package_versions()
    • Run unit tests via python -m pytest apis/python/tests

Bug-fix branches

Problem to solve:

  • TileDB-SOMA 2.n exists, depending on core 2.m et al.
  • We need to bug-fix TileDB-SOMA 2.(n-1), which depends on core 2.(m-1) et al.
  • The R DESCRIPTION-file syntax does not support upper bounds, so any codemod pushed to a non-latest branch will result in the PR on TileDB-SOMA 2.(n-1) getting a version of TileDB-R for core 2.m, unless we explicitly force it not to.
  • Discussion: https://github.com/single-cell-data/TileDB-SOMA/pull/1993#issuecomment-1881150316
  • Solution examples:
    • #1972 pins TileDB-R to 0.21 for the release-1.5 branch
    • #1996 pins TileDB-R to 0.22 for the release-1.6 branch

Using TileDB-SOMA

Developing TileDB-SOMA

Clone this wiki locally