Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Listing blobs is not wrapped in a re-try #743

Closed
rzvoncek opened this issue Apr 11, 2024 · 0 comments · Fixed by #744
Closed

Listing blobs is not wrapped in a re-try #743

rzvoncek opened this issue Apr 11, 2024 · 0 comments · Fixed by #744
Labels
done Issues in the state 'done' enhancement New feature or request

Comments

@rzvoncek
Copy link
Contributor

rzvoncek commented Apr 11, 2024

Project board link

Observed the following exception in the wild:

Traceback (most recent call last):
  File "/home/cassandra/medusa/backup_node.py", line 280, in backup_snapshots
    files_in_storage = storage.list_files_per_table()
  File "/home/cassandra/medusa/storage/__init__.py", line 496, in list_files_per_table
    all_blobs: t.List[AbstractBlob] = self.storage_driver.list_blobs(prefix=fdns_data_prefix)
  File "/home/cassandra/medusa/storage/abstract_storage.py", line 80, in list_blobs
    objects = loop.run_until_complete(self._list_blobs(prefix))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete

This happened because we don't annotate the [list_blobs] with a retry for some reason.

We need to add the retry to protect ourselves from the exception.

@rzvoncek rzvoncek moved this to Ready in K8ssandra Apr 11, 2024
@adejanovski adejanovski added the ready Issues in the state 'ready' label Apr 11, 2024
@rzvoncek rzvoncek moved this from Ready to Ready For Review in K8ssandra Apr 11, 2024
@adejanovski adejanovski added ready-for-review Issues in the state 'ready-for-review' and removed ready Issues in the state 'ready' labels Apr 11, 2024
@rzvoncek rzvoncek added the enhancement New feature or request label Apr 11, 2024
@github-project-automation github-project-automation bot moved this from Ready For Review to Done in K8ssandra Apr 11, 2024
@adejanovski adejanovski added done Issues in the state 'done' and removed ready-for-review Issues in the state 'ready-for-review' labels Apr 11, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
done Issues in the state 'done' enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants