Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Add supports for Byte-Range Fetches when downloading from S3 #2981

Open
gael-ft opened this issue May 23, 2023 · 0 comments · May be fixed by #2982
Open

Add supports for Byte-Range Fetches when downloading from S3 #2981

gael-ft opened this issue May 23, 2023 · 0 comments · May be fixed by #2982
Labels
p:aws-s3

Comments

@gael-ft
Copy link
Contributor

gael-ft commented May 23, 2023

Short description

Add supports for Byte-Range Fetches when downloading an object from AWS S3.

It means doing like AWS "crt" based clients / TransferManager.

Details

Given:

  • bucket
  • S3 key
  • rangeLength (should default to 8MB)
  • parallelism

We should be able to download parallelism ranges in parallel.
Then pushing downstream them in order, requesting requesting a new range when first is complete.

Have my own implementation, I'll submit a PR
@gael-ft gael-ft linked a pull request May 23, 2023 that will close this issue
Open
@ennru ennru added the p:aws-s3 label Jun 16, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
p:aws-s3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

AWS S3: Add getObjectByRanges to S3 API gael-ft/alpakka
2 participants
@ennru @gael-ft