Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Implement a retry mechanism for the FileUploader #2244

Merged
merged 8 commits into from
Jan 28, 2025

Conversation

sfauvel
Copy link
Collaborator

@sfauvel sfauvel commented Jan 24, 2025

Content

Extend the FileUploader trait to support a retry mechanism.

This PR includes:

  • FileUploader trait extension to support a retry mechanism with a blanket implementation.
  • Implement a configurable retry for FileUploaderimplementation
  • Configure all file uploaders of the CardanoDatabase with the default retry values (3 retries and 5 seconds delay)

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • Update README file (if relevant)
    • Update documentation website (if relevant)
    • Add dev blog post (if relevant)

Issue(s)

Closes #2213

Copy link

github-actions bot commented Jan 24, 2025

Test Results

    4 files  ±0     52 suites  ±0   10m 15s ⏱️ -1s
1 553 tests +8  1 553 ✅ +8  0 💤 ±0  0 ❌ ±0 
1 809 runs  +8  1 809 ✅ +8  0 💤 ±0  0 ❌ ±0 

Results for commit 26f5996. ± Comparison against base commit dddc0ee.

♻️ This comment has been updated with latest results.

@sfauvel sfauvel marked this pull request as ready for review January 24, 2025 17:32
@sfauvel sfauvel force-pushed the sfa/2213/implement_fileuploader_retry_decorator branch from a5b6e20 to 5eafafc Compare January 28, 2025 08:27
…he Uploaders and remove `with_retry_policy`
@sfauvel sfauvel force-pushed the sfa/2213/implement_fileuploader_retry_decorator branch from f2c7307 to a99ec30 Compare January 28, 2025 09:39
@sfauvel sfauvel temporarily deployed to testing-preview January 28, 2025 09:48 — with GitHub Actions Inactive
@sfauvel sfauvel temporarily deployed to testing-sanchonet January 28, 2025 09:48 — with GitHub Actions Inactive
@sfauvel sfauvel requested review from Alenar and dlachaume January 28, 2025 09:53
Copy link
Member

@jpraynaud jpraynaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

match self.wrapped_uploader.upload(filepath).await {
Ok(result) => return Ok(result),
Err(_) if nb_attemps >= self.call_limit => {
return Err(anyhow::anyhow!("Upload retry limit reached"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the total number of attempts in this error message?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I proposed: "Upload failed after {} attempts"
Do you have a better suggestion ?

@sfauvel sfauvel temporarily deployed to testing-preview January 28, 2025 10:00 — with GitHub Actions Inactive
@sfauvel sfauvel temporarily deployed to testing-sanchonet January 28, 2025 10:00 — with GitHub Actions Inactive
@sfauvel sfauvel force-pushed the sfa/2213/implement_fileuploader_retry_decorator branch from 4363d83 to 26f5996 Compare January 28, 2025 10:34
Copy link
Collaborator

@dlachaume dlachaume left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Comment on lines +177 to +182
let uploader: Box<dyn FileUploader> = Box::new(LocalUploader::new(
SanitizedUrlWithTrailingSlash::parse("http://test.com:8080/base-root/").unwrap(),
&target_dir,
expected_policy.clone(),
TestLogger::stdout(),
));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of Box doesn't seem necessary.

Suggested change
let uploader: Box<dyn FileUploader> = Box::new(LocalUploader::new(
SanitizedUrlWithTrailingSlash::parse("http://test.com:8080/base-root/").unwrap(),
&target_dir,
expected_policy.clone(),
TestLogger::stdout(),
));
let uploader = LocalUploader::new(
SanitizedUrlWithTrailingSlash::parse("http://test.com:8080/base-root/").unwrap(),
&target_dir,
expected_policy.clone(),
TestLogger::stdout(),
);

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's for have a FileUploader and not a LocalUploader to be sure that retry_policy is the function of the trait.
Otherwise, the test pass even if the trait use another function to retrieve policy.
The test name try to explain that (retry_policy_from_file_uploader_trait_should_be_implemented) but it's probably not enough.
Do you think that we have to add a comment here to explain ?
// Cast into FileUploader to be sure it's the trait function that's being implemented

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I get it, that sounds good to me like that 👍. The test name seems sufficient and clear to me.

@sfauvel sfauvel temporarily deployed to testing-preview January 28, 2025 10:52 — with GitHub Actions Inactive
@sfauvel sfauvel temporarily deployed to testing-sanchonet January 28, 2025 10:52 — with GitHub Actions Inactive
@sfauvel sfauvel merged commit 4ee78f3 into main Jan 28, 2025
46 checks passed
@sfauvel sfauvel deleted the sfa/2213/implement_fileuploader_retry_decorator branch January 28, 2025 13:17
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement a retry mechanism for the FileUploader
4 participants