Add Read::is_append_only method to optimize read_to_string #89711
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've added a new method called `is_append_only` to the `Read` trait. It returns true if `read_to_end` and `read_to_string` can be trusted not to read or overwrite any existing data in their input buffer. These readers use it to signal that they have optimized `read_to_end` methods: * `[u8]` extends the input buffer to the full slice size all at once instead of doubling its size repeatedly until it's big enough. * `File` pre-allocates enough space for the full file size, allowing for fewer `read` syscalls and reducing buffer reallocations. These readers take advantage of it: * If `BufReader` wraps a `[u8]` or `File` its `read_to_string` will call their optimized `read_to_end`s. Overall, this fixes a slow path where `BufReader::read_to_string` would be forced to read into a side buffer when passed a non-empty input buffer.
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
This is what I came up with to fix the slow path identified in #89582 (comment). Introducing a new trait method feels like swatting a fly with a bazooka. I don't have any better ideas, though. (Except, of course, doing nothing.) cc @a1phyr |
jkugelman
commented
Oct 9, 2021
| /// | ||
| /// [`read_to_end`]: Read::read_to_end | ||
| /// [`read_to_string`]: Read::read_to_string | ||
| #[unstable(feature = "append_only", issue = "none")] |
There was a problem hiding this comment.
Placeholder for a new feature / tracking issue.
the8472
added
the
T-libs
|
I'm going to submit a different idea. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've added a new method called
is_append_onlyto theReadtrait. It returns true ifread_to_endandread_to_stringcan be trusted not to read or overwrite any existing data in their input buffer.These readers use it to signal that they have optimized
read_to_endmethods:[u8]extends the input buffer to the full slice size all at once instead of doubling its size repeatedly until it's big enough.Filepre-allocates enough space for the full file size, allowing for fewerreadsyscalls and reducing buffer reallocations.These readers take advantage of it:
BufReaderwraps a[u8]orFileitsread_to_stringwill call their optimizedread_to_ends.Overall, this fixes a slow path where
BufReader::read_to_stringwould be forced to read into a side buffer when passed a non-empty input buffer.