[SURE-9137] ClusterValues dont apply changes if one of the clusters is missing the templateValues

[skanakal]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

If a GitRepo is configured to target two or more clusters and the fleet.yaml file includes ${ .ClusterValues}, any missing templateValues in one of the cluster's spec will prevent updates or changes from being deployed to the clusters where templateValues are properly configured.

Expected Behavior

• The changes should be applied in the cluster where the templatesValues are defined.
• UI should show the clear Error message

Steps To Reproduce

1. Install rancher 2.9.2 with fleet 0.10.3v
2. Register two downstream clusters, ensuring that one of them includes templateValues.

apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  annotations:
  labels:
    foo: bar
    management.cattle.io/cluster-display-name: rke2custom1
    management.cattle.io/cluster-name: c-m-qmc767s2
    objectset.rio.cattle.io/hash: 464bd091084175e4d5572051571f4dfb39bcf2fd
    provider.cattle.io: rke2
  name: rke2custom1
  namespace: fleet-default
spec:
  agentAffinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
              - key: fleet.cattle.io/agent
                operator: In
                values:
                  - 'true'
          weight: 1
  clientID: pl882vs458n4lqqrj8jc58jvkvq4xgqdfv9l7q7spnrhh7s8wjgj8v
  kubeConfigSecret: rke2custom1-kubeconfig
  kubeConfigSecretNamespace: fleet-default
  templateValues:
    generated:
      cluster_metadata:
        fqdn: server-1.example.com
        name: server-1

3. create gitrepo from this example path: templateValues
4. check the gitrepo dashboard for resourceReady

Environment

- Architecture: x86_64
- Fleet Version: fleet:104.0.3+up0.10.3
- Cluster:
  - Provider: custom
  - Options: 1
  - Kubernetes Version: v1.30.5+rke2r1

Logs

From the fleet-controller logs:

2024-10-08T11:59:49Z	DEBUG	bundle	Unchanged bundledeployment	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"mcc-rke2custom1-managed-system-upgrade-controller","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "mcc-rke2custom1-managed-system-upgrade-controller", "reconcileID": "04c5c324-f0f4-4f19-bc31-1e11a890da3e", "bundledeployment": {"apiVersion": "fleet.cattle.io/v1alpha1", "kind": "BundleDeployment", "namespace": "cluster-fleet-default-rke2custom1-43138de7906f", "name": "mcc-rke2custom1-managed-system-upgrade-controller"}, "operation": "unchanged"}
2024-10-08T11:59:49Z	DEBUG	bundle	Unchanged bundledeployment	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"fleet-agent-rke2custom1","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "fleet-agent-rke2custom1", "reconcileID": "d63cdb5d-544d-4356-b269-350b5564aa21", "bundledeployment": {"apiVersion": "fleet.cattle.io/v1alpha1", "kind": "BundleDeployment", "namespace": "cluster-fleet-default-rke2custom1-43138de7906f", "name": "fleet-agent-rke2custom1"}, "operation": "unchanged"}
2024-10-08T11:59:49Z	ERROR	Reconciler error	{"controller": "bundle", "controllerGroup": "fleet.cattle.io", "controllerKind": "Bundle", "Bundle": {"name":"templatevalues-templatevalues-5bfacaa9","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "templatevalues-templatevalues-5bfacaa9", "reconcileID": "2a8aaea7-2194-46c2-a923-bf6f745b1a4a", "error": "failed to render helm values template: template: values:56:40: executing \"values\" at <.ClusterValues.generated.cluster_metadata.fqdn>: map has no entry for key \"generated\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222

Anything else?

current behavior:

[manno]

We should not fail all bundle deployments when one cluster is missing a label.

[p-se]

We should not fail all bundle deployments when one cluster is missing a label.

Clarification: We have decided not to ignore template errors when they occur, but to make them visible in Bundle and GitRepo statuses. Corresponding PR is #3114

[p-se]

/backport v2.10.2

[p-se]

/backport v2.9.6

[weyfonk]

Additional QA

Problem

When a workload targets multiple clusters, and one of those clusters is missing a template value, the following happens:

• the workload is not deployed to any of the target clusters
• a reconcile error appears, but only in fleet-controller pod logs. They are not visible in the Rancher UI.

Solution

Fleet now reflects targeting errors, such as those caused by missing template values on clusters, in the bundle and GitRepo statuses.
Fleet deliberately refrains from creating bundle deployments for clusters without targeting issues. A bundle working with a subset of its expected bundle deployments would be expected to cause inconsistencies in resource counts and a possible cascade of other issues. This could be revisited in a further iteration.

Testing

Engineering Testing

Manual Testing

N/A

Automated Testing

End-to-end tests have been added to check for the presence of targeting errors in bundle and GitRepo statuses.

QA Testing Considerations

Suggestion: follow the reproduction steps above, and check that:

• targeting errors appear in the Rancher UI
• no bundle deployments are created

Regressions Considerations

N/A

[sbulage]

System information

Before Upgrade

Rancher Version	Fleet Version
Prime v2.10.1	0.11.2

Steps used to perform

• Verified that No resources are created on any cluster.
• No Error message

Note: Steps mentioned in the descriptions were performed pre and post upgrade.

---

After Upgrade

Rancher Version	Fleet Version
v2.11-f0b88cfe74d14b4431d11cda695abd69cc1b951d-head	v0.12.0-alpha.3

Steps used to perform

• Upgraded same cluster to v2.11-f0b88cfe74d14b4431d11cda695abd69cc1b951d-head
• Navigate to Continuous Delivery --> GitRepo
• Error message shown on the GitRepo page.
• See below screenshot.
• bundleDeployment for the same GitRepo is not created

Screenshot showing template Error message on GitRepo

---

Below Video shows the Upgrade from Prime v2.10.1 to v2.11-f0b88cfe74d14b4431d11cda695abd69cc1b951d-head version.

cluster_gitrepo_message.mp4

[mahauke]

@sbulage From the screenshot, it is hard to tell whether the cluster with missing values will be visible. Does the Fleet UI show the display name of the cluster which is missing the values?

[sbulage]

@sbulage From the screenshot, it is hard to tell whether the cluster with missing values will be visible. Does the Fleet UI show the display name of the cluster which is missing the values?

Hello @mahauke,
Earlier there were no error shown on the GitRepo page which tell exactly what happens, but right now it is showing due to missing cluster values this GitRepo unable to process further actions.

May be on your specific query @p-se or @manno can shed some light.

[p-se]

@mahauke

@sbulage From the screenshot, it is hard to tell whether the cluster with missing values will be visible. Does the Fleet UI show the display name of the cluster which is missing the values?

If a value is missing or not can only be determined by a GitRepo that points to a repository that uses those cluster values. In fact, the error first appears when the BundleDeployments for the targets are created. Therefore the error is appended to the status of the Bundle. But because we treat the Bundle as a Fleet internal resource, which is not prominently visible in the Rancher UI, the error is propagated to the GitRepo, where you can find it in the UI.

It appears the name of the cluster that is expected to have a value isn't shown. I will revisit the code and check if we can improve on that matter.

[mahauke]

@p-se Thank you for the explanations. It would be great if that could be improved!