Add high throughput integration test #5655
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdettai
commented
Jan 28, 2025
Comment on lines
+93
to
+123
| pub fn merge(self, other: RestIngestResponse) -> Self { | ||
| Self { | ||
| num_docs_for_processing: self.num_docs_for_processing + other.num_docs_for_processing, | ||
| num_ingested_docs: apply_op(self.num_ingested_docs, other.num_ingested_docs, |a, b| { | ||
| a + b | ||
| }), | ||
| num_rejected_docs: apply_op(self.num_rejected_docs, other.num_rejected_docs, |a, b| { | ||
| a + b | ||
| }), | ||
| parse_failures: apply_op(self.parse_failures, other.parse_failures, |a, b| { | ||
| a.into_iter().chain(b).collect() | ||
| }), | ||
| num_too_many_requests: self.num_too_many_requests, | ||
| } | ||
| } |
There was a problem hiding this comment.
I moved this back here as it makes more sense than in the API model because accumulating responses is quite specific to the rest client.
rdettai
force-pushed
the
test-client-retries
branch
4 times, most recently
from
de8f2a1 to
aa98399
Compare
rdettai
force-pushed
the
test-client-retries
branch
2 times, most recently
from
aa77d7e to
c2069e4
Compare
rdettai
force-pushed
the
test-client-retries
branch
from
c2069e4 to
ce4501f
Compare
rdettai
commented
Feb 6, 2025
| // TODO: when using the default 10MiB batch size, we get persist | ||
| // timeouts with code 500 on some lower performance machines (e.g. | ||
| // Github runners). We should investigate why this happens exactly. | ||
| Some(5_000_000), |
There was a problem hiding this comment.
@guilload I didn't find a good explanation for why this timeout occur here in the persist
quickwit/quickwit/quickwit-ingest/src/ingest_v2/router.rs
Lines 432 to 443 in ce4501f
Persisting 10MB should not take 6 sec, even on a slow system and in debug mode, should it?
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is a reuse of the tests and docs proposed in #5644, which itself is not necessary anymore after the the status code was fixed to be 429 when shards need scaling up (#5651).
It also adds a small indication of the number of retries that occurred to the CLI ingest command. This is handy for troubleshooting and shows concretely to users that retries are often necessary.
How was this PR tested?
Integration tests and running the CLI ingest command on the HDFS dataset.