Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Cache mechanism implementation for metadata.json #14224

Conversation

mehmetbutgul
Copy link
Contributor

issue --> #14221
Cache mechanism implementation for metadata.json.
Nowadays, metadata.json is more than 10MB and it will increase in the future.

Description

The main logic is here:
1 - gets3Object that includes getLastModified() (just contains a summary, do not download the whole metadata.json file.)
2- check the condition (cache contains up-to-date metadata)
3- If the cache contains up-to-date metadata, get it;
Otherwise, download it, set it to the cache, and return it.

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Code improvements with no or little impact
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING page.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@mehmetbutgul mehmetbutgul added new-feature Introducing a new feature DON'T MERGE Do not merge this PR labels Apr 3, 2024
@mehmetbutgul mehmetbutgul self-assigned this Apr 3, 2024
@maziyarpanahi
Copy link
Member

I'll let @danilojsl review this as he made the last changes.

@maziyarpanahi maziyarpanahi changed the base branch from master to release/533-release-candidate April 5, 2024 15:38
@maziyarpanahi maziyarpanahi merged commit 2c54a27 into release/533-release-candidate Apr 5, 2024
6 checks passed
@mehmetbutgul mehmetbutgul mentioned this pull request Apr 11, 2024
10 tasks
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
DON'T MERGE Do not merge this PR new-feature Introducing a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cache mechanism is not working related to metadata.json in s3
3 participants