Bump minimum Python version to 3.8 #3204
Conversation
|
I'm 100% in favour of bumping MSRV to 1.56. I'm still open to a decision on whether we think the bump to minimum Python 3.8 needs to happen right away or we should wait for more of the ecosystem to drop 3.7. I realise our policy stated in If we drop 3.7 then it creates more pressure for the rest of the ecosystem to drop 3.7 too, however for projects like @alex, do you have a projection of how long |
adamreichold
force-pushed
the
bump-msrv
branch
2 times, most recently
from
c7fa5ee to
a7efe8f
Compare
|
We don't yet know how long we'll keep 3.7. Our approach to deprecations/removals is based on empirical usage data, not upstream support. Per https://pypistats.org/packages/cryptography, 3.7 is still at nearly 1/4 of our downloads. One of my TODOs is to investigate where those are coming from, likely Debian, in which case perhaps they'll start to go down with the upcoming release. In terms of pinning for 3.7 builds, I looked into this previously, and I could not find a way to potentially depend on multiple different pyo3 versions. |
|
FWIW, if y'all are willing to hold on to 3.7, we (@reaperhulk and I) are willing to do anything required in terms of keeping it working maintenance wise. |
|
Understood; it would be helpful to have a sense of what we're waiting for if we do choose to wait. Just curious - what's the main motivation for not wanting to drop the 3.7 userbase (beyond the notable size)? Releasing a new cryptography which doesn't support 3.7 may help provide them impetus to upgrade.
Indeed, I also took a quick look and came to the same conclusion. |
|
Just had another MSRV failure in #3203; @adamreichold would you be potentially willing to split the MSRV bump into a separate PR which we could land immediately and then we can continue to debate the option of Python 3.7 support in this thread? |
|
Our view is that we have a pretty foundational place in the ecosystem, since we're a dependency of all sorts of things (certbot, twisted, paramiko, etc.). As a result, when we drop something it has pretty wide-ranging impacts -- while old versions do continue to work, they don't get security updates or updates for other ecosystem things. In terms of encouraging people to upgrade, there's definitely a game of chicken and egg. Us dropping 3.7 (or warning about it in advance) does encourage some people to upgrade, however if we're too aggressive, then we impact too large a set of users, at which point maintainers of those various other projects (e.g. certbot) get put in an uncomfortable place, and apply back-pressure to us. For Python 3.6 (which we dropped in our latest release) we started issuing warnings somewhere around 10-20%, and we ultimately dropped it around 7%. So we definitely want to encourage users to upgrade, but at 1/4 of our downloads, it's probably too early to start applying sticks yet, we need things like the new Debian release to come out, and 3.7 to go EOL with upstream, first, then we can come in and apply the sticks to the laggards. FWIW, I pulled the stats on which Linux distros are the source of our 3.7 downloads, and its not what I expected. I don't yet know what the dominance of
|
|
Oh wow. It looks like Amazon Linux 2 does indeed include 3.7 by default if you just install https://aws.amazon.com/amazon-linux-2/faqs/ Amazon Linux 2023 was released on 15th March. Sounds like these are distros derived from Red Hat Linux family with releases planned every two years and support lasting five. With 3.9 shipped as the default in Amazon Linux 2023, we may have a similar position in two years time with 3.9 EOL in Oct 2025 but Amazon Linux 2023 continuing until 2028. https://aws.amazon.com/linux/amazon-linux-2023/faqs/ So this 50% of 3.7 downloads does have an upgrade path already, and Amazon is committed to pushing security back ports for them (they have a |
|
Oops, I just noticed I left a sentence half written: I was not expecting Amazon Linux to be such a dominant thing, and I don't yet have a sense for what it means for our Python 3.7 support. @reaperhulk and I will need to discuss. |
Sure, but the bump seems to be more complicated than expected insofar there is a lot Clippy-related work to be done, i.e. I am not sure when I will be able to do this. Concerning Python 3.7, what do you think about dropping support from PyO3's mainline but committing to keep maintaining 0.19.x upstream? cryptography could then stick to 0.19.x which should not be an issue functionality-wise as we do not support anything crossing extensions yet. |
|
I have a strong preference for keeping 3.7 support on main -- this keeps us engaged with upstream and contributing back. As I said, we're happy to take on any work needed to keep it functional. |
I think the main issue is that this work if often poorly defined and not separable from the rest of the work, i.e. random things fail because of some incompatibility with 3.7 and the code is more complicated as the various workarounds have to stay in place. And OTOH, if you do make changes to 0.19.x, porting them forward to main and making the necessary adjustments is well-defined and has limited chance of interference. Obviously, I see the downside that it can end up being twice the work.
@davidhewitt One solution to this would be to do check builds or at least allow warnings for the MSRV clippy CI job. I actually think that check builds would be ideal because we want to be Clippy-clean on stable, but we just want to build on MSRV. Especially since being Clippy-clean on stable and MSRV can actually conflict. |
Agreed with this, if you're short on time I'm happy to give this a go tonight. |
Would be glad if you looked into it as I won't be able to do anything about it tonight. |
|
Have pushed #3208 for the MSRV bump, which looks like it is going to pass. As for the Python bump, I'm very torn. As a maintainer I would be glad to see 3.7 go; our CI is already extremely heavy and we'll soon be adding more to it with 3.12. I also think that it's likely that reworks we add for interpreter initialization will not support 3.7 (which looks necessary for part of work to support 3.12 as per #2709 (comment)). We could just not offer the reworked API on 3.7, though it's a bit unfortunate. On the other side of the coin, I'm very keen to help PyO3 extension authors support 3.7 if they feel that there is good reason to. I want PyO3 to be seen as a useful part of the ecosystem that authors can rely on, so if axing 3.7 support too fast will isolate extension authors to be stuck on older versions of PyO3, that seems like a marketing downside to me. Especially when we haven't yet added 3.12 support and I'm hopeful we'll also land other big improvements over the second half of this year. What would we do here if PyO3 were past 1.0? I think we would keep the original plan of doing MSRV bumps and Python minimum bumps together, as both of them would be cause for a major release (rather than just a minor release with feature additions). @alex, I'm keen to hear the outcome of the evaluation of Amazon Linux 2 impact on 3.7 support between yourself and @reaperhulk (but no need to rush, 0.20 won't be released for perhaps a month or so). |
3208: bump msrv to 1.56 r=davidhewitt a=davidhewitt The MSRV changes from #3204, plus a commit which should hopefully make CI pass. With luck this is mergeable and resolves CI pain while we decide what to do about the Python version. Co-authored-by: Adam Reichold <adam.reichold@t-online.de> Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>
I think that depends on the marketing. If we do commit to maintaining multiple PyO3 versions for increases compatibility? Yes, packages will not get both the newest PyO3 features and compatibility with 3.7, but we could keep existing environments alive as long as they are in use with repeatedly facing hard choice between keeping the code base healthy and keeping it compatible. |
|
I haven't had a chance to talk with @alex much about this yet, but we've recently come off the 0.15.x branch (we made it to 0.18.x with our latest release). During our time on 0.15.x we needed the pypy3.9 compatibility work that was backported into 0.15.2 (thanks to everyone on that!). Additionally, we recently got an issue filed about pypy3.10, which would have been problematic again except we were able to move to 0.18.x just in time. 😄 Loss of forward compatibility is definitely something we definitely worry about. An official policy on supporting a long-lived branch (with some metrics for when that support ends) might assuage that concern. That said, I wonder if the maintenance required to keep an older release up-to-date and forward compatible will exceed keeping that support in Thinking more broadly, having a primary release (where you get new APIs, features, etc) that isn't compatible with the set of versions in wide use increases the adoption cost for projects looking to switch to PyO3. Do you choose the newer features or do you support all the versions your users are asking for? That's a decision each project will need to make as part of evaluating PyO3 if you prioritize dropping support. Regardless of how you choose, I want to express my thanks for the volunteers on this project. We love that we've been able to ship safer code to our users for the past 2+ years (and well over 1B downloads) and a huge amount of credit goes to this project for making that possible. |
Personally, I am rather convinced that yes, forward porting will be more work quantitively. I just think that it would keep "compatibility work" from interfering with "feature work." For example, I have people contributing rarely in mind which I think often hit these complications while doing something else entirely.
Indeed we would be pushing that choice out into the ecosystem somewhat which admittedly is not a nice thing to do. The main argument for doing this that I can see is that a specific project with specific goals is in a better to position to make an informed decision. We as an upstream project would need to make everyone happy though which notably includes trying to grow our contributor and probably even more important reviewer base. |
adamreichold
changed the title
|
As an alternative idea how about: 3.7 jobs are allowed to fail on PRs (i.e., they can be merged), and then we can get ping'd to resolved the issues asynchronously? |
|
That's an interesting proposal and may be an acceptable compromise. Our I'm just trying to enable GH merge queue in #3206 - so I'm in the process of learning if it is possible for 3.7 to be failing but the merge queue to pass. Downsides I see of allowing 3.7 to fail on merge:
|
|
If you use something like https://github.com/re-actors/alls-green to set a list of required jobs that must pass then the merge queue should work fine with potentially failing jobs. You can see how we use all-green in our ci.yml, although we just use auto-merge, not the queue feature. |
|
Yes, that's right, any sort of issues required to keep 3.7 green (post-hoc) we could ping'd on and we'd do our best to resolves in a timely manner. |
|
Closing this as the consensus seems to be that we keep support for Python 3.7 on the main branch. |
|
At least for 0.20, I'm willing to see how this works out. If it turns out to be a lot of bother for everyone, let's reconsider dropping. And hopefully users of cryptography will hurry up and update soon 😅 |
|
Thanks! |
This is not trying to reap any of the benefits of making that change yet, just the minimum amount of bureaucracy to make it happen.