Skip to content

Branches and releases

Jump to bottom
John Kerl edited this page Sep 17, 2024 · 58 revisions

Things everyone should know

  • Everyone should commit to main as a general rule
  • Micros are tagged on branches such as release-1.5, release-1.6, release-1.7
    • The first will be 1.5.0, 1.6.0, 1.7.0, etc
    • If there are subsequent PRs on a branch we can make a 1.5.1, 1.6.1, etc.
  • When you merge a PR to main, simply add a tag backport release-1.5, backport release-1.6, etc. This will put up a PR for that branch.
  • Timing:
    • Recall major.minor.micro: the minor is the second number and the micro is the third.
    • I know of no scenario which would cause core's major to bump, nor TileDB-SOMA's.
    • Whenever core bumps a minor, dependent packages including TileDB-SOMA bump a minor.
      • Example: core 2.19 appears, TileDB-Py 0.25 is tagged depending on 2.19, TileDB-R 0.23 is tagged depending on 2.19, then a TileDB-SOMA PR can be put up (on main) depending on 2.19, 0.25, and 0.23
      • When core 2.20 appears (along with TileDB-Py 0.26 and TileDB-R 0.24) we will tag a TileDB-SOMA 1.8 -- regardless of how much or how little development work has happened in the TileDB-SOMA repo since 1.7.
    • If it so happens that there is a significant new feature in TileDB-SOMA, we reserve the right to bump the TileDB-SOMA minor even if there hasn't been a core release yet. As of January 2024, this hasn't happened yet, but it could.
    • When core bumps a micro -- e.g. 2.19.0 to 2.19.1 -- we may or may not bump our micro.
      • If the core change doesn't affect TileDB-SOMA at all, we might have no need to bump our micro.

Things the release engineer should know

Roles

  • TileDB-SOMA is an open-source collaboration between TileDB and CZI
  • Most of this checklist, except the Conda and TileDB-Cloud steps, can be done by someone at either company
  • Someone at TileDB needs to be involved to complete the Conda and TileDB-Cloud steps

Incrementing core versions

Example scenario:

  • Currently TileDB-SOMA depends on core 2.20, TileDB-Py 0.26, and TileDB-R 0.24.
  • The latest release branch is release-1.8.
  • Now a 2.21 has been released.
  • We need to:
    • Have main depend on 2.21, 0.27, and 0.25
    • Create a release-1.9 branch
    • Ensure that release-1.8 continues to get TileDB-R 0.24 (now that 0.25 is existing)

What needs to be done to make that happen:

  • For a PR on the main branch:
    • Run scripts/update_tiledb_version.py 2.21.0 (or whatever version)
    • Edit apis/python/setup.py to bump the TileDB-Py dependency to 0.27 (or whatever version)
    • Edit apis/r/DESCRIPTION to bump the TileDB-R dependency to 0.25 (or whatever version)
    • All that should go into a single PR like https://github.com/single-cell-data/TileDB-SOMA/pull/2150
  • For a PR on the latest release branch (release-1.8) in this example:
  • After all of this is stable in main, you can make a release-1.9 branch (or whatever version) following the procedure below in this document

What to wait for:

If we don't wait for all of those to be true -- if there's some urgency in getting the release out before all R binaries are available -- we need to do things like this:

and then this change will need to be undone in a separate PR a day or two later once all the above binary artifacts are available.

Deciding on a release

  • Post in the common Slack channel that you want to do a release
    • Include a list of the PRs that will be in the release — the GitHub UI lets you auto-generate this list
    • Wait for two people to agree
  • Validate the version number and wait for two people to agree
    • We use semantic versioning which is seemingly unambiguous: majors for backward-incompatible changes, minors for new features that are backward-compatible, micros for bugfixes
    • But in reality this is subjective: for example, if something was a bit incomplete before, was its incompleteness a bug being fixed, or is the new bit a new feature?
    • Be sure to also bump the tiledbsoma version at apis/r/DESCRIPTION, as this lives outside of GitHub versioning
  • Considering the scope of changes in the release, a release candidate (e.g. i.j.krc0, with no - before rc0, for PEP386) might be required if:
    • If new dependencies, end-to-end validation is required
    • If new features are introduced, validation of features is required
    • If there are removals of APIs, validation of downstream software is required
    • If there are deprecations, proper release notes with explanations and replacements outlines is required
    • If a release candidate was created, all validations must be completed and successful to cut the release

Validating

  • CI in the TileDB-SOMA repo is our #1 signal
  • Conda CI (below)
  • Notebook evaluations
    • At first, manually evaluate them
    • Over time: automate notebook-evaluations in a CI job

Branch/tag mechanics

conda install -n root -c conda-forge conda-smithy
# or perhaps
mamba install conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

# or keeping your base env clean
mamba create --yes -n rerender conda-smithy
conda activate conda-smithy
cd tiledbsoma-feedstock
conda smithy rerender --commit auto

If

RuntimeError: conda-smithy version (3.24.1) is out-of-date (3.26.3) in conda-smithy's environment. Exiting.

then

mamba upgrade -c conda-forge conda-smithy -n base

and retry the above.

Artifacts

Note that all Conda artifacts for dependent packages must be present in order to Conda-ify TileDB-SOMA.

Aside for arm64 (M1/M2) Macs

  • As noted on https://github.com/single-cell-data/TileDB-SOMA/pull/1385, using GitHub Actions as of mid-2023 the best we can do for MacOS arm64 is to cross-compile for arm64 while actually running on x86_64.
  • This is tracked here: https://github.com/github/roadmap/issues/528 -- TL;DR maybe 2023Q4 GHA will have this available
  • This also means we can build the arm64 wheels in CI but we cannot smoke-test them -- since we do not have arm64 hardware to run the smoke test on
    • Consolation prize:
    • Get the artifacts from the CI job
    • Download and unzip wheels-macos-arm64.zip
    • pip install tiledbsoma-i.j.k-cp311-cp311-macosx_11_0_arm64.whl (or whatever your OS version)
    • In Python, import tiledbsoma and tiledbsoma.show_package_versions()
    • Run unit tests via python -m pytest apis/python/tests

Bug-fix branches

Problem to solve:

  • TileDB-SOMA 2.n exists, depending on core 2.m et al.
  • We need to bug-fix TileDB-SOMA 2.(n-1), which depends on core 2.(m-1) et al.
  • The R DESCRIPTION-file syntax does not support upper bounds, so any codemod pushed to a non-latest branch will result in the PR on TileDB-SOMA 2.(n-1) getting a version of TileDB-R for core 2.m, unless we explicitly force it not to.
  • Discussion: https://github.com/single-cell-data/TileDB-SOMA/pull/1993#issuecomment-1881150316
  • Solution examples:
    • #1972 pins TileDB-R to 0.21 for the release-1.5 branch
    • #1996 pins TileDB-R to 0.22 for the release-1.6 branch

Using TileDB-SOMA

Developing TileDB-SOMA

Clone this wiki locally