Ease migration of experimental to stable packages

[lzchen]

Cont'd discussion from 01/21/21 SIG

Taken from https://github.com/open-telemetry/opentelemetry-python/blob/master/rationale.md
Our current planned release versioning story is as follows:

V1.0.0 Release (tracing, baggage, propagators, context)
opentelemetry-api 1.0.0
Contains APIs for tracing, baggage, propagators, context
opentelemetry-tracing 1.0.0
Contains the tracing SDK
opentelemetry-sdk 1.0.0
Contains SDK components for tracing, baggage, propagators, and context
Contains the following experimental packages
opentelemetry-api-metrics 0.x.y
Contains the EXPERIMENTAL API for metrics. There are no stability guarantees.
opentelemetry-metrics 0.x.y
Contains the EXPERIMENTAL SDK for metrics. There are no stability guarantees.
V1.15.0 Release (with metrics)
opentelemetry-api 1.15.0
Contains APIs for tracing, baggage, propagators, context, and metrics
opentelemetry-tracing 1.15.0
Contains tracing SDK
opentelemetry-metrics 1.15.0
Contains metrics SDK
opentelemetry-sdk 1.15.0
Contains SDK components for tracing, baggage, propagators, context and metrics

In terms of following the specs and having backwards compatibility when releasing, we are good. The only issue is when we move experimental packages from experimental to our stable api package, there is a point of contention for customers. With the current design, we expect users to use opentelemetry-api. They can install opentelemetry-tracing and use that to get tracing/baggage/propagator/context functionality or the can install opentelemetry-sdk which will simply just pull in opentelemetry-tracing. If users want to try want to try out experimental metrics signals, they can pull in both opentelemetry-api-metrics and opentelemetry-metrics. These packages will ONLY have experimental related signals.

Now when we release v1.15.0, the experimental api signals will be moved into the stable opentelemetry-api package. Ideally there should be no changes to the namespace, so users should not have to modify any of their import paths to begin using metrics (if they were using the experimental package previously). The only thing that they have to do is to uninstall the experimental opentelemetry-api-metrics package to avoid conflicts.

Now the question is. Would this step of "removing the opentelemetry-api-metrics package" be a problem for customers? I can see how customers might miss this step and have to go to the community for help. However, with good documentation, we can mitigate this problem. As well, this would only affect customers that were trying out the experimental signals. Another downside is there would be a package floating around after v1.15.0 (opentelemetry-api-metrics) that we would need to deprecate.

Thoughts on this?

[owais]

I liked @ocelotl's idea of having a single package with a stable and dev version. So we'd release opentelemetry-api 1.0 and it'd only contain tracing and other stable components. At the same time, we'd also release opentelemetry-api 1.1-dev0 which would contain metrics and any other unstable components. The idea being that a future release (1.1.0) might ship metrics and other unstable components if they stablize in time for release. If they don't, then we ship 1.1.0 again without metrics and at the same time ship 1.2.0-dev0 with metrics. So unstable components keep moving to n+1-dev package until they stablize. This way users can install the dev release of a future version if they want to unstable components.

This would be more work for us but as a user this would be most intuitive (follows what other projects do), least surprising (can't have both installed at the same time, won't break due to automatic minor version updates).

[lzchen]

@owais
With this solution, how would a user be able to use the stable components and try out the unstable components at the same time if both packages are the same?

[owais]

Technically they wouldn't be able to do that. If they chose to try out the dev package, by definition, the entire package including the tracing bits would be consider unstable and subject to change. However, I think this shouldn't be too much of a problem given we don't intent to make any significant changes to the tracing API and I think any minor API changes are acceptable if users chose to use a dev version as it means they'd be open to making a few changes every now and then anyway.

There is precedent for this sort of packaging almost everywhere. For example, if I'm using Django 3.0 but want to try the shiny new ORM feature in 3.1-dev before 3.1 is released, I have to be open to using in development version of the views, middlewares, templates or any other sub-component. I think it's an acceptable trade-off to have. As a user, if I'm too concerned about updates breaking my services, I'd chose to run stable Otel for tracing and another stable solution for metrics. On the other hand, if I'm brave and excited enough to run unstable Otel for metrics, I don't think the minor additional risk of tracing API breaking would be a great deal to me.

[lzchen]

@owais
So would opentelemetry-api 1.1-dev0 contain metrics + unstable components AND all the stuff in opentelemetry-api 1.0 as well?

[codeboten]

i think the option with a dev and stable version probably works better, since in the future we'll want to be able to have an API package that may be unstable (2.0) at which point we would have a need for this anyways.

[ocelotl]

@owais
With this solution, how would a user be able to use the stable components and try out the unstable components at the same time if both packages are the same?

To try the unstable components, it is just a matter of creating a new virtual environment and trying it out 🙂

[lzchen]

@ocelotl
I think my point was that the stable package and unstable package are the same, just simply different versions. So if the unstable package ONLY contains the unstable components, users wouldn't be able to use both stable and unstable at the same time within the same venv.

[ocelotl]

@ocelotl
I think my point was that the stable package and unstable package are the same, just simply different versions. So if the unstable package ONLY contains the unstable components, users wouldn't be able to use both stable and unstable at the same time within the same venv.

Hmm, the unstable package will always be a "working" version in the sense that it has everything needed to work with spans, metrics, etc. What happens is that it includes newer features that we still don't feel confident enough to call stable or that they may not be completely specified, etc. Usually the unstable package will be a super set of the all the stable features and the unstable ones. I say usually because when we do non-backwards compatible changes (if we ever do that) the unstable package may not contain stable features.

[lzchen]

@ocelotl
Yup I agree. Just was confused because what @owais said before "At the same time, we'd also release opentelemetry-api 1.1-dev0 which would contain metrics and any other unstable components.". If the unstable package is a SUPERSET then I have no problem with this approach.

I say usually because when we do non-backwards compatible changes (if we ever do that) the unstable package may not contain stable features.

I think as discussed with @codeboten , if users take a dependency on the unstable version, they are taking a risk on possibility of breaking changes. I think the promise from us will be, there will be no breaking changes (we will maintain backwards compatibility) between STABLE versions of the package.

[aabmass]

(follows what other projects do)

@owais could you share some projects that do this? I don't see any -devX releases for django on PyPI. I am curious how to achieve this with setuptools without copying the code to two separate package directories or branches.

IIRC this approach won't work with "compatible release" specifier (~=) to get patch releases. I haven't tested this, but I don't think opentelemetry-api~=1.1.0-dev0 would match opentelemetry-api~=1.1.1-dev0. Users basically have to pin the version.

If a pre-release, post-release or developmental release is named in a compatible release clause as V.N.suffix, then the suffix is ignored when determining the required prefix match

[aabmass]

I want to throw something else out and see what everyone thinks. We could have 4 packages:

• opentelemetry-api
• opentelemetry-api-experimental
• opentelemetry-sdk
• opentelemetry-sdk-experimental

The experimental packages would have all of the current metrics things under a namespace package opentelemetry.experimental.metrics/opentelemetry.experimental.sdk.metrics. They would be versioned < 1.0.0 but can have regular version numbers like 0.X.Y.

In opentelemetry-api (and likewise for sdk), the existing opentelemetry/metrics/__init__.py file could be replaced with

import * from opentelemetry.experimental.metrics

and we update setup.cfg options.extra_requires with an optional dependency for metrics where we can fully specify the compatible opentelemetry-api-experimental (and sdk) versions.

So if users want metrics, all they have to do is pip install opentelemetry-api[experimental-metrics]. Users would do from opentelemetry import metrics to use metrics. If they don't install the experimental packages, they get an import error.

When metrics go GA, we move the code out of the experimental packages into the api/sdk packages (and rewrite the internal imports to remove references to opentelemetry.experimental) which replace the stub __init__.py files. Users don't have to change their code and the experimental packages can live on with any other experimental stuff. The options.extra_requires would just be an empty list and we can remove it later.

[ocelotl]

(follows what other projects do)

@owais could you share some projects that do this? I don't see any -devX releases for django on PyPI.

@aabmass

Here it is: https://pypi.org/project/Django/3.2a1/
Here is the latest stable version of Django: https://pypi.org/project/Django/3.1.5/

It seems like they have matching tags: https://github.com/django/django/tags

[codeboten]

Just confirming, @lzchen and I tested the following:

• created a release at 0.17.0
• created a following release at 0.17.1b0 (which pypi automatically marks as pre-release)

pip installed the package with a ~= 0.17 requirement

result
the 0.17.0 package was installed and the 0.17.1b0 was ignored.

you can see the release history here: https://test.pypi.org/project/opentelemetry-opentracing-shim/0.17.0/#history

i think based on this test that a dev package suffix would work as intended here. thoughts @aabmass?

[aabmass]

@codeboten that makes sense. What I'm concerned about is that users can use the pre-release/dev version and still get patches and new minor versions without manually changing the requirement. E.g. if I pip install django~=3.0 I get the latest compatible stable release Django==3.1.5. But if I try pip install django~=3.0rc1 I get 3.0rc1, not the latest compatible minor version RC, which would be 3.1rc1.

I think the only way to do always stay on the dev release would be pip install --pre "opentelemetry==1.*" ..., but then pip would choose pre-releases for all packages.

[owais]

I think my point was that the stable package and unstable package are the same, just simply different versions. So if the unstable package ONLY contains the unstable components, users wouldn't be able to use both stable and unstable at the same time within the same venv.

I'm sorry. I should've said it'll contain unstable components in addition to the stable ones.

What I'm concerned about is that users can use the pre-release/dev version and still get patches and new minor versions without manually changing the requirement. E.g. if I pip install django~=3.0 I get the latest compatible stable release Django==3.1.5. But if I try pip install django~=3.0rc1 I get 3.0rc1, not the latest compatible minor version RC, which would be 3.1rc1.

I think this is isn't bad and is how I'd expect it to work. I think a package automatically "jumping" from one pre-release version to the next release's pre-release version would be very surprising and unexpected. I see what you're saying though. You want users to be able to tell their package manager to always fetch the latest release that has metrics in it. I see value in that but it feels like a very special case if you look at it purely from package/release management perspective. IMO, it's fine to have users manually upgrade to next pre-release version. This would also ensure apps don't break on their own on a simple re-deploy just because we released the next pre-release version. If we really wanted to make this happen, we could just release a pre-release version of the eventual stable version that we think would ship with metrics and keep bumping the dev version number. For example, if we think metrics will become stable in 1.7, then we release 1.7-dev0 now and just keep incrementing the dev part. Eventually we'll reach 1.7 stable and it'll have metrics. I don't think it's worth doing though but happy to hear thoughts.

[owais]

To clarify, I think someone wanting to always and automatically stay on the latest pre-release version of a package irrespective of the major.minor version is extremely rare and I don't think we should try to account for that. If someone really wants to do that, it can be automated with a simple bash script or a makefile target. For most users, it makes a lot more sense if such updates don't happen automatically on every deploy.

[lzchen]

Okay so I'm seeing some leaning in favor of what @owais has referred to [here](#1547 (comment)?

Sample of what this would look like:

V1.0.0 Release (tracing, baggage, propagators, context)
opentelemetry-api 1.0.0 Contains APIs for tracing, baggage, propagators, context
opentelemetry-sdk 1.0.0 Contains SDK components for tracing, baggage, propagators, and context
Contains the following experimental releases
opentelemetry-api 1.1.0-dev0 Contains api for tracing, baggage, propagators, context and metrics. There are no stability guarantees.
opentelemetry-sdk 1.1.0-dev0 Contains sdk for tracing, baggage, propagators, context and metrics. There are no stability guarantees.

V1.15.0 Release (with metrics)
opentelemetry-api 1.15.0
Contains APIs for tracing, baggage, propagators, context, and metrics
opentelemetry-sdk 1.15.0
Contains SDK components for tracing, baggage, propagators, context and metrics
Contains the following experimental releases
opentelemetry-api 1.2.0-dev0 Contains api for tracing, baggage, propagators, context, metrics and other unstable signals. There are no stability guarantees.
opentelemetry-sdk 1.2.0-dev0 Contains sdk for tracing, baggage, propagators, context, metrics and other unstable. There are no stability guarantees.

Customers would simply install the stable version of the api/sdk once they are deemed stable.

Please react to this post if you are okay with it.

[ocelotl]

@lzchen

To prevent misunderstandings, I say we detail this plan a bit more specifically, something like this:

0. We will follow a release pattern similar to Django.
1. We will have one opentelemetry-api package in PyPi
2. This package will have several releases in PyPi (Like Django does)
3. This package will always have one release marked as latest which is the stable one (Like Django does (notice the green "Latest version" mark))
4. This package will have zero or more unstable releases (Like Django does (notice the yellow warning that points to the stable version))
5. Users should only have to change the specified version of the installed package to switch from stable to unstable or to any other release.
6. Users should never need to make any modification in application code to switch from the installed version of any opentelemetry package.
7. ...

It does not have to be exactly like this, but I think it is important to always specify what do we mean with "opentelemetry-x". From the previous conversation in the SIG I noticed that it is easy to get confused since when we say opentelemetry-api, it may refer to the package in PyPi, the path from opentelemetry import x, the actual code, etc.

[ocelotl]

To clarify, I think someone wanting to always and automatically stay on the latest pre-release version of a package irrespective of the major.minor version is extremely rare and I don't think we should try to account for that. If someone really wants to do that, it can be automated with a simple bash script or a makefile target. For most users, it makes a lot more sense if such updates don't happen automatically on every deploy.

I agree with @owais here. Having a way to always install the latest unstable release can be definitely useful but I don't find it to be a main goal of this releasing system. Also, if our users feel so adventurous, they can always use a Git path that points to master in their requirements.txt file 😄

[aabmass]

To clarify, I think someone wanting to always and automatically stay on the latest pre-release version of a package irrespective of the major.minor version is extremely rare and I don't think we should try to account for that.

I agree automatically staying on latest pre-release is generally unwanted, which is why I think releasing metrics in pre-releases for each version is not ideal. From a contrib perspective, it's also kind of weird having all of the exporters/instrumentations that support metrics have a specific pre-release in their install_requires (or will these packages be split?).

---

I still don't understand how our source code would look, could someone explain. Our pre-releases are essentially being cut from a different source set than the true releases if I understand right. Is that really a pre-release?

There is precedent for this sort of packaging almost everywhere. For example, if I'm using Django 3.0 but want to try the shiny new ORM feature in 3.1-dev before 3.1 is released, I have to be open to using in development version of the views, middlewares, templates or any other sub-component.

This isn't the same as what is proposed above though: "So unstable components keep moving to n+1-dev package until they stablize." I can start updating my Django application against 3.1-dev in preparation for Django 3.1, but Django 3.1 isn't a moving target. There is an implication that the features/milestones in the dev release will make it into the RC and then into the release. Generally, just bugs are being worked out in the release.

[aabmass]

Also want to point out that companies that use a mono-repo (like Google 😉) typically only have exactly one version of OSS packages used in the whole codebase. If we go with this approach, many such companies will be forced to either 1) never wait to try metrics or 2) use an unstable dev version everywhere. This defeats the purpose of calling traces 1.0 imo.

[ocelotl]

Also want to point out that companies that use a mono-repo (like Google ) typically only have exactly one version of OSS packages used in the whole codebase. If we go with this approach, many such companies will be forced to either 1) never try metrics or 2) use an unstable dev version everywhere. This defeats the purpose of calling traces 1.0 imo.

So, Googlers can't use the unstable releases of Django? 🤔

[ocelotl]

Here is more information regarding Django release process.

[owais]

Also want to point out that companies that use a mono-repo (like Google 😉) typically only have exactly one version of OSS packages used in the whole codebase. If we go with this approach, many such companies will be forced to either 1) never wait to try metrics or 2) use an unstable dev version everywhere. This defeats the purpose of calling traces 1.0 imo.

The purpose of having 1.0 release is that people who want just tracing can now start using OpenTelemetry with a guarantee that it won't break in future. This allows users to migrate away from OpenTracing and OpenCensus for tracing. Users can continue to use OpenCensus or Prometheus for metrics. Users brave enough to use unstable APIs can opt in to use OpenTelemetry metrics as well by using a pre-release package. This serves the purpose perfectly IMO.

Also, no 1.x.y pre-release package, pre-release or not, should break the tracing API. Once 1.0 is released, we cannot break anything in 1.x.y but only add new things. If we decide to break anything released in 1.0.0, we must release 2.0 so this shouldn't be a problem until we releease 2.0.0-dev right? So if a breaking change to the tracing API is not scheduled for a 1.x.y release then it'd never make it to 1.x.y-dev either. So in practice, you should still have stable tracing API when using any pre-release version in the 1.x series.

[aabmass]

@owais I guess it depends how we structure the code and deep we are willing to go with it. If we have a dev branch where we merge non-compatible (next major version) tracing + metrics things and a stable branch that we merge fixes/patches (minor and patch versions), then sure. But it might become burdensome regularly merging the stable fixes/patches into the dev branch, especially after any refactoring.

How will the code/repo structure and maintenance look for this scheme?

[lzchen]

@ocelotl

To prevent misunderstandings, I say we detail this plan a bit more specifically, something like this:

Yes I agree with the points you stated above. I think just to add a one more point as discussed above:
8. Compatible installs (~=) will not install pre-release versions. Pre-release versions must be explicitly specified.

@aabmass
I talked about this with @codeboten and we discussed having the possibility of a "master" branch, in which will contain the stable releases and a "dev" branch, which will contain the non-compatible changes that you mentioned above.

I think the idea behind this is that "master" (being the default branch), will always contain release ready code. We will then instruct contributors to make PRs against the "dev" branch. We will merge features them into master as we deem the ready to be released. I think eventually when we can mark all signals as stable, we will no longer have a need for a dev branch, as all changes from then on should be bug fixes/patches.

The benefits of this is it will be easier for maintainers (bug fixes can go into master, unstable signals go into dev). The downside is that it will require a bit of onboarding for contributors to make PRs against a specific branch.

We will also have our GH actions setup to run tests, builds, etc for the new dev branch as well.

But it might become burdensome regularly merging the stable fixes/patches into the dev branch, especially after any refactoring.

Speaking about the other way (merging bug fixes/patches INTO the dev branch) should be simple manually. I was planning on manually doing this on a weekly basis or whenever there was a significant change in master. If this proves to be too cumbersome, maybe we can visit the idea of adding something into GH actions.

[owais]

Alternatively, we could have just one branch but with the help of some scripts, include metrics only in the dev releases. This could totally be automated. That said master and dev branches is how this is done normally. I was just throwing things out there. Not necessarily suggesting we do it.

[lzchen]

@owais
It would be great if this was automated, but given our lack of time and resources, we may need to go with the multiple branch approach for the time being. As well, I could see it being an execution nightmare, where there will be special cases in new features in which automation isn't smart enough. Regardless, I think the burden for maintainers is not too great, especially if we manage only two branches, and as well, if we will eventually get rid of the dev branch.

[aabmass]

I seem to be the only one with a differing opinion 😅 in which case I am ok with whatever.

Speaking about the other way (merging bug fixes/patches INTO the dev branch) should be simple manually. I was planning on manually doing this on a weekly basis or whenever there was a significant change in master. If this proves to be too cumbersome, maybe we can visit the idea of adding something into GH actions.

@lzchen I think it would get cumbersome with a lot of merge conflicts (same ones over and over again). Usually the dev branch would be getting merged into master each release so it's not a huge issue, but here we are waiting on the metrics timeline.

E.g. PR1 to the dev branch renames a variable. PR2 is a fix to the master branch, using the old variable name; you go to cherry pick that PR into dev (or merge master into dev) and you will get a conflict. Later, PR3 to the dev branch implements a new (backward-compatible) tracing feature with the new variable name; you get a similar conflict when cherry picking or merging that into master.

Even if you made a GH action, it should catch the change in PR1 but that is impossible because PR2 is in the future.

[lzchen]

@aabmass
Yup that's definitely a fair use case. I believe once we are in 1.0.0 there should be less and less fixes into the master branch and more into new features in dev. It might be a little complicated as you say in the beginning but I think the benefits outway the short term cons here.

@ocelotl @owais @codeboten
Are we all in concensus with the above approach with details specified and execution?

[ocelotl]

@aabmass
Yup that's definitely a fair use case. I believe once we are in 1.0.0 there should be less and less fixes into the master branch and more into new features in dev. It might be a little complicated as you say in the beginning but I think the benefits outway the short term cons here.

@ocelotl @owais @codeboten
Are we all in concensus with the above approach with details specified and execution?

I agree 👍