Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline #336

Closed
VladNF opened this issue Feb 14, 2020 · 6 comments
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. status: investigating The issue is under investigation, which is determined to be non-trivial. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@VladNF
Copy link

VladNF commented Feb 14, 2020

Certain cloud-ndb queries causes grpc to spawn threads that prevent accomplishing webapp2/django requests as GAE standard env for python 2 joins all request threads.
Requests failes with the following exception

Thread running after request. Creation traceback:
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 80, in Run
    target(*args, **kw)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 794, in __bootstrap
    self.__bootstrap_inner()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 821, in __bootstrap_inner
    self.run()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 774, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_channel.py", line 678, in channel_spin
    event = state.completion_queues[key].poll()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_plugin_wrapping.py", line 107, in __call__
    wrapped_context, AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/base/data/home/apps/.../lib/google/auth/transport/grpc.py", line 87, in __call__
    future = self._pool.submit(self._get_authorization_headers, context)
  File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 137, in submit
    self._adjust_thread_count()
  File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 159, in _adjust_thread_count
    t.start()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 749, in start
    _start_new_thread(self.__bootstrap, ())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 83, in StartNewThread
    return base_start_new_thread(Run, ())

The following shows one example that after re-writing it in a get-by-id manner has no such issue.

Model1.query(Model1.repeatedKeyProp.IN(keys)).fetch()

Shutting down ThreadPoolExecutor from concurrent.future helped to localize the issue caused by additional threads caused by querying data.

Any ideas how to coupe with it? We cannot just move everything to Python 3 env in a one step, both envs are going to co-exist for sometime.

@VladNF VladNF changed the title [GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests to hang over the deadline [GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline Feb 14, 2020
@andrewsg
Copy link
Contributor

Thanks for this report. To clarify, in this case your query was fired and completed, and results delivered, entirely within the request; then, at the end of the request, there was an error? Or is the problem that the query was not completed at all and errored out immediately, even though the request had not completed?

@VladNF
Copy link
Author

VladNF commented Feb 15, 2020

Yes, a query is completed and results are delivered. The things is AppEngine runtime joins all threads spawned within a request, somehow those additional threads after query with IN,AND etc are not finished. I tried to figure out what are these threads and seems like they are from grpc concurrent.future.thread.ThreadPoolExecutor. In our code we do not start any background thread, just using cloud-ndb to handle data (worked fine with AppEngine built-in ndb module before migration to cloud-ndb).

@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Feb 15, 2020
@oliverchang
Copy link

We also ran into this (google/clusterfuzz#1298 (comment)).

The workaround for us is to manually invoke the GC at the end of the request, which seems to clean up stray threads.

@VladNF
Copy link
Author

VladNF commented Feb 20, 2020

@oliverchang thank you so much, seems to work for us

@VladNF VladNF closed this as completed Feb 20, 2020
@chrisrossi chrisrossi reopened this Feb 20, 2020
@chrisrossi
Copy link
Contributor

Reopening this so we can investigate whether it makes sense to do something on the NDB side.

@oliverchang mentions this happens with MultiQuery but not other calls? Is it worth exploring why?

Would it make sense to add a call to gc.collect somewhere in NDB?

@chrisrossi chrisrossi added priority: p2 Moderately-important priority. Fix may not be included in next release. status: investigating The issue is under investigation, which is determined to be non-trivial. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed 🚨 🚨 This issue needs some love. triage me I really want to be triaged. labels Feb 20, 2020
chrisrossi pushed a commit to chrisrossi/python-ndb that referenced this issue Mar 20, 2020
This works around an issue where a thread started by `google.auth`
confuses the Google App Engine Python 2.7 runtime.

Fixes googleapis#336
@chrisrossi
Copy link
Contributor

This is fixed in google.auth with googleapis/google-auth-library-python#467

The gc.collect() work-around should no longer be required.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. status: investigating The issue is under investigation, which is determined to be non-trivial. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants