Figure out what's causing Python 3.11 instability

[zorbathut]

We have a problem where Python 3.11 is causing uncommon crashes on the live server. Some stacktrace screenshots:

Things tried:

• Changing the base image to ubuntu:22.04 works fine on Python 3.10. Updating to Python 3.11 is difficult because Poetry isn't compatible with the Deadsnakes 3.11 (Poetry is incompatible with unstable-tagged Python builds (Invalid PEP 440 version: '3.Y.Z+') python-poetry/poetry#6925). This should be tested further.
• poetry:3.10 works fine; poetry:3.11 doesn't work.
• I haven't gotten this to happen in the test framework, although the test framework doesn't do real HTTP requests. Maybe we should try making real HTTP requests?
• This happens both on the live site and the dev site.
• This has not been observed locally, however.
• I've tried recompiling sqlalchemy without its C module. This made no difference.
• I've tried disabling gevent, greenlet, and the messy signal stuff we're doing. This made no difference.

If you've got ideas on how to reproduce this, let me know, I'm happy to try stuff out.

[justcool393]

this appears to be some sort of memory corruption of some sort. given the random nature of where the crashes are occurring and the fact that these crashes are occurring in both code that makes no sense to cause them and some other stack traces where the top frame was "Garbage collecting..." i'm relatively convinced of this.

python 3.11 iirc is notable for performance improvements and if i were to guess (no evidence for this) is that something expected something to be somewhere in python 3.10 but it's not there in python 3.11 and this is causing memory corruption.

given this i'm inclined to believe our culprit is one of (in descending order of likelihood)

1. c extensions. there are a bunch of things we use C for indirectly and someone who isn't playing nice by using the correct memory allocation functions might be opening it up.
2. python 3.11 itself. i find this unlikely, but i do think py3.11 is a factor.
3. some random freak of nature that hates us specifically. this is prolly it tbh.

[justcool393]

oddly enough one of the strange things with this is that it's hard for me to reproduce. i'm curious does prod + dev have any major differences with deployment than the docker version?

we probably could try and point valgrind at it to see what's blowing up but we'd need a coredump for that