Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

bucket: Add prefix filtering capability #1228

Merged
merged 3 commits into from
Oct 17, 2023
Merged

bucket: Add prefix filtering capability #1228

merged 3 commits into from
Oct 17, 2023

Conversation

stefanprodan
Copy link
Member

@stefanprodan stefanprodan commented Sep 8, 2023

This PR adds a new optional field .spec.prefix to the Bucket API. When specified the prefix is used to filter the objects server-side when .spec.provider is set to generic, aws and gcp.

Fix: #693
Fix: #248

Testing

if you want to try out this feature on your own test cluster:

  1. Install the latest Flux controllers
flux install
  1. Apply the Bucket CRD from this branch
kubectl apply --server-side --field-manager=flux -f https://raw.githubusercontent.com/fluxcd/source-controller/bucket-prefix/config/crd/bases/source.toolkit.fluxcd.io_buckets.yaml
  1. Deploy the source-controller build from this branch
kubectl -n flux-system set image --field-manager=flux deployment/source-controller manager=ghcr.io/fluxcd/source-controller:bucket-prefix-8e215264

@stefanprodan stefanprodan added enhancement New feature or request area/bucket Bucket related issues and pull requests labels Sep 8, 2023
@tiffanywang3
Copy link

Thank you @stefanprodan; using the updated source controller image + Bucket CRD, we were able to leverage the prefix filtering which reduced Bucket reconciliation time from ~2.5 min to ~6 seconds (generic provider).

Looking forward to this update getting included in next Flux release!

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
@stefanprodan stefanprodan requested a review from hiddeco October 17, 2023 11:36
@stefanprodan stefanprodan marked this pull request as ready for review October 17, 2023 11:36
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
@stefanprodan stefanprodan added the area/api API related issues and pull requests label Oct 17, 2023
@stefanprodan stefanprodan merged commit f2a1814 into main Oct 17, 2023
11 checks passed
@stefanprodan stefanprodan deleted the bucket-prefix branch October 17, 2023 12:24
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
area/api API related issues and pull requests area/bucket Bucket related issues and pull requests enhancement New feature or request
Projects
None yet
3 participants