RuntimeError: cannot schedule new futures after interpreter shutdown for python 3.9+

[konradzagozda]

Current Behavior

I got this stack trace when I wanted to follow SDK intro using python 3.9 or 3.10 - resulting in event not being sent

/home/konrad/PycharmProjects/pythonProject10/venv/bin/python /home/konrad/PycharmProjects/pythonProject10/main.py 
Consumer thread error
Traceback (most recent call last):
  File "/home/konrad/PycharmProjects/pythonProject10/venv/lib/python3.10/site-packages/amplitude/worker.py", line 76, in buffer_consumer
    self.threads_pool.submit(self.send, events)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Process finished with exit code 0

Possible Solution

There are workarounds like these https://stackoverflow.com/questions/70673175/runtimeerror-cannot-schedule-new-futures-after-interpreter-shutdown
but I think I should be able to follow introduction without going into these

Steps to reproduce

1. create new project with python 3.9 3.10
2. install amplitude analytics
3. paste into main:

from amplitude import *


client = Amplitude('785b2b6680dd0758b2ca0511441dd9e5');

client.track(BaseEvent(
    event_type='Python Event',
    user_id='datamonster@gmail.com',
    ip='127.0.0.1',
    event_properties={
        'keyString': 'valueString',
        'keyInt': 11,
        'keyBool': True
    }
))

4. run

Environment

• SDK Version: amplitude-analytics==1.1.0
• Python Version: 3.9 or 3.10
• OS Info: Ubuntu 20.04

I don't know if this is related to the way I run the script? Does that mean python 3.9 and 3.10 are not supported?

[liuyang1520]

Thanks for reporting this! I think we definitely support 3.9 and 3.10. I tried in my local with 3.9.2, everything seems working fine. Could you post the minor version of 3.9 mentioned in the environment? I will try the 3.10 too.

[konradzagozda]

I didn't mention in the issue, it worked for Python <= 3.8

my version of python 3.9:

(venv) konrad@konrad-ubuntu:~/PycharmProjects/pythonProject11$ python --version
Python 3.9.14
(venv) konrad@konrad-ubuntu:~/PycharmProjects/pythonProject11$

I could reproduce it for 3.9.2:

# Dockerfile
FROM python:3.9.2
WORKDIR /usr/app/src
COPY main.py ./
COPY requirements.txt ./
RUN pip install -r requirements.txt

CMD [ "python", "./main.py"]

docker image build -t ampli .
docker image run ampli

(venv) konrad@konrad-ubuntu:~/PycharmProjects/pythonProject11$ docker run ampli
Consumer thread error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/amplitude/worker.py", line 76, in buffer_consumer
    self.threads_pool.submit(self.send, events)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 163, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
(venv) konrad@konrad-ubuntu:~/PycharmProjects/pythonProject11$ 

Changing image version to 3.8 fixes issue

[yeutterg]

I can confirm the issue @konradzagozda is having.

On Python 3.9.14 and 3.10.8, I get the error "RuntimeError: cannot schedule new futures after interpreter shutdown"

On Python 3.7.14 and 3.8.14, the library runs fine.

This is occurring on an M1 Pro MacBook Pro running MacOS 13. Also happened on MacOS 12. Amplitude-Python version 1.1.0.

Full error readout:

 File "/Users/greg/.pyenv/versions/3.9.14/lib/python3.9/site-packages/amplitude/worker.py", line 76, in buffer_consumer
    self.threads_pool.submit(self.send, events)
  File "/Users/greg/.pyenv/versions/3.9.14/lib/python3.9/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

[liuyang1520]

Hi @konradzagozda , @yeutterg , thanks for the feedback!

I tried both Python 3.9.2 and 3.10.8, both are working fine without triggering the above issue. Here are my steps:

Flask app

• create venv with python version 3.9.2 or 3.10.8
• cd examples/flask_example
• pip install amplitude-analytics
• update api key in flaskapp.py file
• pip install flask
• FLASK_APP=flaskapp flask run
• Trigger the event with http://127.0.0.1:5000/pageloaded multiple times in browser

Python script

• create venv with python version 3.9.2 or 3.10.8
• cd examples/track_example
• pip install amplitude-analytics
• update api key in trackevent.py file
• python3 trackevent.py
• Event is sent automatically

My env:

• MacOS 12.5.1
• M1 Max chip

If this is related to your instrumentation, is it possible to share a minimal example here for us to reproduce the issue?

Thanks!

[bohan-amplitude]

Hi @konradzagozda @yeutterg ,

Could you help verify if you can add a client.shutdown() or client.flush() call before application exit? This method call can schedule a task to threads pool to empty the events buffer. Optionally add a short period time (time.sleep(0.1)) after call shutdown() to give interpreter thread grace period wait the last task scheduled to threads pool.

[ssbarnea]

@bohan-amplitude I have python 3.11 and I got the same errors, is this SDK ever passed testing with versions of python that did not reach end-of-life?

Based on https://github.com/amplitude/Amplitude-Python/actions/runs/3473700633 i really believe that the only version it was tested with is 3.6 which is already EOL and deprecated, pending removal even from github actions.

When is @amplitude going to implement a testing pipeline that is testing with all current python versions, we already have 5 versions that are not tested with 3.7-3.11

[qingzhuozhen]

I could reproduce this and seems this is an issue happening in scripts only, and only for Python version 3.9+.

The runtime error is thrown at cpython/thread.py at 3.9 · python/cpython. In submit method, it checks global variable _shutdown and if that is true the program will throw the error. From Python 3.8 to 3.9, the exit register way also changed in this PR. I can also see reports after this PR this error starts happening more. This change continues to the most current version and hasn’t changed since then.

The reason for this is, in scripts run, when the main thread completes it will call the exit method, and mark the _shutdown to be true. While in SDK we are submitting events after the interval delay. This issue now is only affecting scripts run and not affecting usage in the backend server since the main thread didn’t close there.

The fix for this is tricky. I tried to use the same way in thread.py file to register a method on exit, inside just call flush. That works for Python 3.9. However, that method is not available under Python 3.9, considering the version we support starts from Python 3.6, we can’t do that now. I also tried the module method atexit in Python as an alternative, however, that is giving me the same error in Python 3.9. It seems so far there is not a good and quick way to resolve this script-specific issue. (unless we consider changing the way of threads pool and rewriting the whole threading logic, which means a much higher investment. Welcome any suggestions or any other thoughts for this.)

To temporarily alleviate this issue, I would suggest calling the flush method at the end of the program. Call the shutdown method also works, though that means it is closing resources. Another way to resolve this is to block the main thread a bit with sleep and let the program finishes.

Please let us know if these could help and feel free to share more feedback or suggestions.

[qingzhuozhen]

Following up on the previous context, I put a temp workaround solution for this at #43. Feel free to share any feedback on it. Tested this on Python 3.9 and seems working fine for the script case.

[qingzhuozhen]

We just released v1.1.1 including the above fix. Feel free to upgrade and let us know if the experience is improved or not. Thanks for everyone's feedback!

[simon-weber]

After upgrading to 1.1.1 we didn't see this error for a while, but unfortunately we just got two instances of it. The stacktrace is unchanged (modulo the line numbers moving around a little bit).

It seems like it's still an issue, but less common now.

[qingzhuozhen]

Hi @simon-weber, thanks for the feedback. Would you mind sharing more details about how to reproduce in your case? We can dive more into it. Thank you.

[simon-weber]

I don't think I've got anything too insightful to share, unfortunately. It's running under gunicorn/django on Heroku, and the errors seem to happen when dynos get shut down.

[justin-fiedler]

Hi @simon-weber we are still unable to reproduce this issue. I am going to close but please reopen if you can provide steps to reproduce. Thank you.

Possible issues:

1. the script does not terminate immediately - it can wait for self.storage.lock.wait(...) up to flush_interval_millis (10 seconds by default) in function Workers.buffer_consumer
   Amplitude-Python/worker.py at main · amplitude/Amplitude-Python
2. flush() sends all pending events, flush_queue_size is ignored here (Amplitude-Python/worker.py at main · amplitude/Amplitude-Python ). Request payload can be huge.
3. Are you using a custom Destination plugin? Could there be errors in that logic?