Replace upload mechanism to use a batched mutation with a new API #888
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uploadbuild-mutation' into ghengeveld/ap-3971-cli-pass-content-hashes-to-uploadbuild-mutation
…o-uploadbuild-mutation
…e finishing upload task
…s-to-handle-thousands-of-files
…-thousands-of-files' into ghengeveld/ap-3912-index-update-cli-to-wait-for-copy-job-using-sentinel-file
ghengeveld
changed the title
uploadBuild mutation, running in batches
ghengeveld
changed the title
ghengeveld
changed the title
…-cli-to-wait-for-copy-job-using-sentinel-file Retrieve `sentinelUrls` from `uploadBuild` and wait for all of them before finishing upload task
ghengeveld
deleted the
ghengeveld/ap-4034-batch-file-upload-requests-to-handle-thousands-of-files
branch
|
🚀 PR was released in |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This reintroduces the
uploadBuildmutation after it was reverted. It's now limited to sending 1000 files at a time in order to prevent issues with request/response payload size and/or memory limits. To accommodate that constraint, files are now batched in chunks of 1000 anduploadBuildruns in a loop.Because S3 does not support uploading empty files directly, those are skipped and a warning is printed. The
zipoption can be used to work around this limitation.The new
uploadBuildmutation provides a more flexible API so it can support file deduplication.Depends on https://github.com/chromaui/chromatic/pull/8134
📦 Published PR as canary version:
10.2.3--canary.888.7540838602.0✨ Test out this PR locally via:
npm install chromatic@10.2.3--canary.888.7540838602.0 # or yarn add chromatic@10.2.3--canary.888.7540838602.0