Release v0.12.0 #235
Merged
Release v0.12.0 #235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [highlevel] Stop using memory-mapped files While investigating this SDK for uploading and downloading large blobs (e.g. 25GB or more) it became apparent that the memory-mapped approach has some severe limitations: 1. Limits the file size on 32-bit systems (theoretically 4GB, but much less in practice). 2. Has no backpressure when writing to slower storage mediums. 3. Appears to finish faster, but the OS spends several minutes flushing the modified RAM to disk afterwards (depends on the speed of the disk). On a VM with 16GB of RAM and a slow disk (spinning in this case) the algorithm quickly overwhelms the available memory and causes severe performance degradation. It ended up simultaneously trying to flush to the slow data disk and page out to the slightly faster OS disk. The solution is to stop using memory-mapped files (at least the way the SDK currently uses then) and switch to the `io.ReaderAt` and `io.WriterAt` interfaces. They explicitly allow for parallel access to non-overlapping regions which make them a good candidate for this purpose. Benchmarking large downloads (25GB file) between azcopy 10.4.3 and these updates using a test app, the difference between them is within 10 seconds. When compared against the original code on a beefy machine with plenty of RAM the measured execution time is faster, but there is a little bit of delay while the last of the data flushes from RAM to disk. * PR feedback
The condition that tests the HTTP status code to decide whether or not to upgrade log severity never sees the 4xx and 5xx responses because they are encapsulated in a `StorageError`. Addresses #214
Correctly handle HTTP errors in RequestLogPolicy
The patch that tried to de-encapsulate the response status code assumed the error would be of the `StorageError` type. We now test the error type before trying to get to the response; if the error is not a `StorageError`, the log severity will always be increased.
Issue/http log policy
* Basic CPK-V/CPK-N Code * Minor Fix * Rev - 1 | Fixed Comment * Added CPK in retry download
* v0.12.0 Release ChangeLogs & BreakingChanges * Updated changelogs * Added mmf changes in ChangeLog.md
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.