Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Leverage batch delete in vacuum for GCS #407

Closed
Dandandan opened this issue Aug 22, 2021 · 4 comments
Closed

Leverage batch delete in vacuum for GCS #407

Dandandan opened this issue Aug 22, 2021 · 4 comments
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed storage/gcp

Comments

@Dandandan
Copy link
Contributor

Dandandan commented Aug 22, 2021 •
edited
Loading

Description

GCS supports batching multiple delete operations (up to 100) in a single call. This can speed up vacuum.
We shuld add a new delete_objs implementation of StorageBackend for GCS.

Use Case

Related Issue(s)
#394
@Dandandan Dandandan added the enhancement New feature or request label Aug 22, 2021
@houqp houqp added binding/rust Issues for the Rust crate good first issue Good for newcomers help wanted Extra attention is needed storage/gcp labels Aug 22, 2021
@houqp
Copy link
Member

houqp commented Aug 22, 2021

Thanks @Dandandan for filing this, FYI @blogle in case you run into vacuum performance issue with your PoC.

@blogle
Copy link
Contributor

blogle commented Aug 23, 2021

I haven't had the need to do any work around vacuum just yet, and to be honest I have been thinking I would just invoke vacuum on some schedule from spark when needed.... As we further scope out our needs and ensure those features are built out, I wouldn't mind going through and adding this to the storage backend's, assuming nobody does so in the interim.

@wjones127
Copy link
Collaborator

Now that we use objectstore, we should implement this upstream: apache/arrow-rs#2615

@rtyler
Copy link
Member

rtyler commented Oct 25, 2023

Closing this since we have this support now, and we're improving it further with the adoption of delete_stream

@rtyler rtyler closed this as completed Oct 25, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed storage/gcp
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@rtyler @Dandandan @houqp @blogle @wjones127