-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
[GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline #336
Comments
Thanks for this report. To clarify, in this case your query was fired and completed, and results delivered, entirely within the request; then, at the end of the request, there was an error? Or is the problem that the query was not completed at all and errored out immediately, even though the request had not completed? |
Yes, a query is completed and results are delivered. The things is AppEngine runtime joins all threads spawned within a request, somehow those additional threads after query with IN,AND etc are not finished. I tried to figure out what are these threads and seems like they are from grpc concurrent.future.thread.ThreadPoolExecutor. In our code we do not start any background thread, just using cloud-ndb to handle data (worked fine with AppEngine built-in ndb module before migration to cloud-ndb). |
We also ran into this (google/clusterfuzz#1298 (comment)). The workaround for us is to manually invoke the GC at the end of the request, which seems to clean up stray threads. |
@oliverchang thank you so much, seems to work for us |
Reopening this so we can investigate whether it makes sense to do something on the NDB side. @oliverchang mentions this happens with Would it make sense to add a call to |
This works around an issue where a thread started by `google.auth` confuses the Google App Engine Python 2.7 runtime. Fixes googleapis#336
This is fixed in The |
Certain cloud-ndb queries causes grpc to spawn threads that prevent accomplishing webapp2/django requests as GAE standard env for python 2 joins all request threads.
Requests failes with the following exception
The following shows one example that after re-writing it in a get-by-id manner has no such issue.
Shutting down ThreadPoolExecutor from concurrent.future helped to localize the issue caused by additional threads caused by querying data.
Any ideas how to coupe with it? We cannot just move everything to Python 3 env in a one step, both envs are going to co-exist for sometime.
The text was updated successfully, but these errors were encountered: