[backport v2.9] [SURE-9487] Fleet takes long time to create a bundledeployment (180+ gitrepos)

[rancherbot]

This is a backport issue for #3163, automatically created via GitHub Actions workflow initiated by @aruiz14

Original issue body:

SURE-9487

Issue description:

At a larger gitrepo scale a applying a new gitrepo using labels takes a long time to create the bundledeployment

Repro steps:

• create a huge number of gitrepos. (k apply -f gitrepos.yaml) gitrepos.yaml is attached to the case
• create one extra git repo that has targets using labels

  paths:
    - /hello-world
  repo: https://github.com/rbreddy/bundledependency.git
  targets:
    - clusterSelector:
        matchLabels:
          foo: bar

• watch the resources :

kubectl get clusters -n fleet-default --output-watch-events --watch-only | ts '%Y-%m-%d %H:%M:%S' | tee  -a output.txt 
kubectl get bundles -n fleet-default --output-watch-events --watch-only | ts '%Y-%m-%d %H:%M:%S' | tee -a output.txt

• Label a cluster to get a gitrepo targeting it:

kubectl label cluster.management.cattle.io <CLUSTER-ID> -n fleet-default foo=bar --overwrite | ts '%Y-%m-%d %H:%M:%S' | tee -a output.txt 

*watch resources on the DS cluster: e.g.

kubectl get pods -n hello --watch | ts '%Y-%m-%d %H:%M:%S' | tee -a output.txt

• Actual behavior:
  fleet takes time to create bundle deployments and start creating resources

• Expected behavior:
  fleet should immediately create bundle deployments and start creating resources

• Additional notes:
  With 300 gitrepos it takes 2 minutes

Click to expand

2024-12-04 15:48:57 cluster.management.cattle.io/c-m-hz9bmvz5 labeled 2024-12-04 15:48:57 EVENT NAME CLUSTERCLASS PHASE AGE VERSION 2024-12-04 15:48:57 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:48:57 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:48:57 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:48:57 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:29 EVENT NAME BUNDLEDEPLOYMENTS-READY STATUS 2024-12-04 15:50:29 MODIFIED bundledependency-hello-world 1/2 Pending(1) [Cluster fleet-default/test1] 2024-12-04 15:50:30 NAME READY STATUS RESTARTS AGE 2024-12-04 15:50:30 nginx-helloworld-67d799d946-xl94f 0/1 ContainerCreating 0 0s 2024-12-04 15:50:30 nginx-helloworld-67d799d946-xl94f 0/1 Pending 0 0s 2024-12-04 15:50:30 nginx-helloworld-67d799d946-xl94f 0/1 Pending 0 0s 2024-12-04 15:50:31 nginx-helloworld-67d799d946-xl94f 0/1 ContainerCreating 0 1s 2024-12-04 15:50:33 nginx-helloworld-67d799d946-xl94f 1/1 Running 0 3s 2024-12-04 15:50:38 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:38 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:38 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:38 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:50:39 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:53:28 MODIFIED test1 Provisioned 2d4h 2024-12-04 15:54:04 MODIFIED bundledependency-hello-world 2/2

• with only one git repo it starts the next second

Click to expand

2024-12-09 14:20:37 cluster.management.cattle.io/c-m-hz9bmvz5 labeled 2024-12-09 14:20:37 EVENT NAME BUNDLEDEPLOYMENTS-READY STATUS 2024-12-09 14:20:37 EVENT NAME BUNDLEDEPLOYMENTS-READY STATUS 2024-12-09 14:20:37 EVENT NAME CLUSTERCLASS PHASE AGE VERSION 2024-12-09 14:20:37 EVENT NAME CLUSTERCLASS PHASE AGE VERSION 2024-12-09 14:20:37 MODIFIED bundledeployment-hello-world 0/1 Pending(1) [Cluster fleet-default/test1] 2024-12-09 14:20:37 MODIFIED bundledeployment-hello-world 0/1 Pending(1) [Cluster fleet-default/test1] 2024-12-09 14:20:37 MODIFIED bundledeployment-hello-world 0/1 WaitApplied(1) [Cluster fleet-default/test1] 2024-12-09 14:20:37 MODIFIED bundledeployment-hello-world 0/1 WaitApplied(1) [Cluster fleet-default/test1] 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:37 MODIFIED test1 Provisioned 7d2h 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Deployment does not have minimum availability., Available: 0/1 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Deployment does not have minimum availability., Available: 0/1 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Deployment does not have minimum availability., Replicas: 0/1 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Deployment does not have minimum availability., Replicas: 0/1 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Replicas: 0/1 2024-12-09 14:20:38 MODIFIED bundledeployment-hello-world 0/1 NotReady(1) [Cluster fleet-default/test1]; deployment.apps hello/nginx-helloworld [progressing] Replicas: 0/1 2024-12-09 14:20:38 NAME READY STATUS RESTARTS AGE 2024-12-09 14:20:38 NAME READY STATUS RESTARTS AGE 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 ContainerCreating 0 0s 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 ContainerCreating 0 0s 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 Pending 0 0s 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 Pending 0 0s 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 Pending 0 0s 2024-12-09 14:20:38 nginx-helloworld-67d799d946-lgrmt 0/1 Pending 0 0s 2024-12-09 14:20:39 nginx-helloworld-67d799d946-lgrmt 0/1 ContainerCreating 0 1s 2024-12-09 14:20:39 nginx-helloworld-67d799d946-lgrmt 0/1 ContainerCreating 0 1s 2024-12-09 14:20:41 MODIFIED bundledeployment-hello-world 1/1 2024-12-09 14:20:41 MODIFIED bundledeployment-hello-world 1/1 2024-12-09 14:20:41 nginx-helloworld-67d799d946-lgrmt 1/1 Running 0 3s 2024-12-09 14:20:41 nginx-helloworld-67d799d946-lgrmt 1/1 Running 0 3s

/home/mmartin/Downloads/gitrepos-300.yaml

Environment:

Rancher Cluster:
Rancher version: 2.9.4
Number of nodes: 3
Node OS version:

Downstream Cluster:
Number of Downstream clusters: ~50
Node OS:
RKE/RKE2/K3S version: AKS
Kubernetes version: 1.28