Python API: rework EC upload code #1814
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The previous code may have considered as successes some uploads that had failed.
AymericDu
approved these changes
Jun 17, 2019
kamel-rahim
approved these changes
Jun 17, 2019
fvennetier
changed the title
AymericDu
approved these changes
Jun 18, 2019
Rework the EC upload code to make it more readable. Also, parallelize the upload of the trailers, so a slow chunk upload does not block the termination of the other chunk uploads.
fvennetier
force-pushed
the
count-successes
branch
from
4b0c957 to
bdeb43d
Compare
|
LGTM |
fvennetier
changed the title
fvennetier
force-pushed
the
count-successes
branch
from
97bdf46 to
9f3b06b
Compare
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
The previous code may have considered as successes some uploads that had failed (removing elements from the list we are iterating on is a bad thing). Now we build a list with only the chunks which do not see any error yet.
Also, parallelize the upload of the trailers, so a slow chunk upload does not block the termination of the other chunk uploads.
ISSUE TYPE
COMPONENT NAME
SDS VERSION