Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Improve documentation for multiprocessing/threading around the boto3.client method. #2991

Open
nateprewitt opened this issue Jul 30, 2023 · 3 comments
Labels
documentation This is a problem with documentation. feature-request This issue requests a feature. p2 This is a standard priority issue

Comments

@nateprewitt
Copy link
Contributor

nateprewitt commented Jul 30, 2023

Currently, invoking boto3.client to create a client calls create_client on the default Boto3 session. This can lead to concurrency issues when done across parallelization primitives unexpectedly. We should update the documentation to clarify both in the client and session docs that calling this method inside threads may not be safe. The safe approach is to create a client in the parent thread/process either with the default session or a user instantiated session. This client can be shared as needed.

Client Documentation
Session Documentation

@nateprewitt nateprewitt added feature-request This issue requests a feature. needs-triage This issue or PR still needs to be triaged. labels Jul 30, 2023
@nateprewitt nateprewitt changed the title Improve documentation for multiprocessing around the boto3.client method. Improve documentation for multiprocessing/threading around the boto3.client method. Jul 30, 2023
@tim-finnigan tim-finnigan added documentation This is a problem with documentation. p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Jul 31, 2023
@mustafaelghrib
Copy link

Hi, @nateprewitt I am interested in this and I will work on it. I have some questions:

  • Is that should be opened in the boto3 repo instead? As the docs you mention are there and I will open a pull request in that repo not here. Should I open another issue in the boto3 and mention this issue or just make the pull request directly?
  • Do you want to make it appear like a note, a normal text, or something else?
  • Should I put what you have written in a little format in the note? or do you have another better description?

@nateprewitt
Copy link
Contributor Author

nateprewitt commented Aug 17, 2023

Hi @mustafaelghrib,

Is that should be opened in the boto3 repo instead?

That's correct that the changes needed here will be done in the boto3 repository. Specifically here in the guides section for client and session.

You can reference this issue in your boto3 PR, I wouldn't worry about creating a duplicate there.

Do you want to make it appear like a note, a normal text, or something else?

A small note or single sentence at the end of each section is likely sufficient.

Should I put what you have written in a little format in the note? or do you have another better description?

This would be a reasonable snippet for the note:

"Note that boto3.client uses a single, shared session for all calls. This can lead to concurrency issues unexpectedly when done across parallelization primitives. We recommend managing your own session(s) with concurrent code."

@mustafaelghrib
Copy link

mustafaelghrib commented Aug 17, 2023

Okay @nateprewitt I opened a pull request, you could check it now

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
documentation This is a problem with documentation. feature-request This issue requests a feature. p2 This is a standard priority issue
Projects
None yet
Development

No branches or pull requests

3 participants