NullPointerException in SSABasedGenericKubernetesResourceMatcher

[kosmoz]

Bug Report

What did you do?

We ran into this issue when using unsupported beta features on an older Kubernetes version:

• Create a StatefulSet dependent resource with persistentVolumeClaimRetentionPolicy on a cluster running Kubernetes version 1.26.
• Kubernetes adds persistentVolumeClaimRetentionPolicy to the managedFields but ignores the value
• When getting the StatefulSet, the SDK expects all managed fields to be present in the resource but in this case, persistentVolumeClaimRetentionPolicy is missing.

What did you expect to see?

At most, a warning should be printed.

What did you see instead? Under which circumstances?

Error during event processing ExecutionScope{ resource id: ResourceID{name='redacted', namespace='redacted'}, version: 8966444451} failed. 
io.javaoperatorsdk.operator.AggregatedOperatorException: Exception(s) during workflow execution. Details:
 - eu.glasskube.operator.apps.vault.dependent.VaultStatefulSet -> java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because "actualMap" is null
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:144)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.matches(SSABasedGenericKubernetesResourceMatcher.java:90)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:169)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:32)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:67)
	at io.javaoperatorsdk.operator.processing.dependent.SingleDependentResourceReconciler.reconcile(SingleDependentResourceReconciler.java:19)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:52)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.doRun(WorkflowReconcileExecutor.java:115)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.NodeExecutor.run(NodeExecutor.java:22)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowResult.throwAggregateExceptionIfErrorsPresent(WorkflowResult.java:41)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileResult.throwAggregateExceptionIfErrorsPresent(WorkflowReconcileResult.java:9)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.DefaultWorkflow.reconcile(DefaultWorkflow.java:95)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:110)
	at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:109)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:140)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:121)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:409)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Environment

Kubernetes cluster type:

vanilla

$ Mention java-operator-sdk version from pom.xml file

4.4.2

$ java -version

openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7, mixed mode, sharing)

$ kubectl version

Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.4
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

Possible Solution

Check if actualMap is null before adding to the result.

Additional context

Related: glasskube/operator#267

[shawkins]

Relates to #2028 - it's the same NPE that happens when using a secret with stringData in the desired state.

[csviri]

Hi @kosmoz , on short notice you might want to try the generic matcher, see:
https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java#L19-L19

also:
https://javaoperatorsdk.io/docs/v4-4-migration#using-server-side-apply-in-dependent-resources

Would be great if you could provide a simple project to reproduce. Or provide the logs from here: https://github.com/shawkins/java-operator-sdk/blob/c879806ba279c2a285639344a7e34c01118c8c01/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/SSABasedGenericKubernetesResourceMatcher.java#L86-L86

thank you!

[kosmoz]

Hi! I'm not sure if I will have time to create a minimal reprodicution repo. However the steps would be pretty straight forward:

1. create an operator with one controller for a CRD
2. create a managed dependent resource for that CRD. This should be a StatefulSet where persistentVolumeClaimRetentionPolicy is set to something other than null.
3. On a cluster running a Kubernetes version <1.27 create a resource for that CRD

Maybe I will have time next week, but in the meantime, here are the logs (I pruned them to the lines that I think matter for this error).

2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] TRACE i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - Original actual: 
 StatefulSet(apiVersion=apps/v1, kind=StatefulSet, metadata=ObjectMeta(annotations={}, creationTimestamp=2023-08-25T09:00:45Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}), manager=vaultreconciler, operation=Apply, subresource=null, time=2023-08-25T09:00:45Z, additionalProperties={}), ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}), manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z, additionalProperties={})], name=vault-test, namespace=default, ownerReferences=[OwnerReference(apiVersion=glasskube.eu/v1alpha1, blockOwnerDeletion=null, controller=null, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9, additionalProperties={})], resourceVersion=2242, selfLink=null, uid=33102e14-0069-4979-8739-64f86f00822e, additionalProperties={}), spec=StatefulSetSpec(minReadySeconds=null, ordinals=null, persistentVolumeClaimRetentionPolicy=null, podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector=LabelSelector(matchExpressions=[], matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}, additionalProperties={}), serviceName=vault-test-headless, template=PodTemplateSpec(metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[], name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[EnvVar(name=SKIP_SETCAP, value=true, valueFrom=null, additionalProperties={}), EnvVar(name=HOSTNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=SUBDOMAIN, value=vault-test-headless, valueFrom=null, additionalProperties={}), EnvVar(name=POD_IP, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=status.podIP, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_API_ADDR, value=https://$(POD_IP):8200, valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201, valueFrom=null, additionalProperties={}), EnvVar(name=DB_HOST, value=vault-test-db-rw, valueFrom=null, additionalProperties={}), EnvVar(name=DB_NAME, value=vault, valueFrom=null, additionalProperties={}), EnvVar(name=DB_USERNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=username, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=DB_PASSWORD, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=password, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME), valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_K8S_POD_NAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_K8S_NAMESPACE, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.namespace, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={})], envFrom=[], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle=Lifecycle(postStart=null, preStop=LifecycleHandler(exec=ExecAction(command=[killall, vault], additionalProperties={}), httpGet=null, tcpSocket=null, additionalProperties={}), additionalProperties={}), livenessProbe=null, name=vault, ports=[ContainerPort(containerPort=8200, hostIP=null, hostPort=null, name=https, protocol=TCP, additionalProperties={}), ContainerPort(containerPort=8201, hostIP=null, hostPort=null, name=https-internal, protocol=TCP, additionalProperties={})], readinessProbe=null, resources=ResourceRequirements(claims=[], limits={memory=100Mi}, requests={memory=30Mi}, additionalProperties={}), securityContext=SecurityContext(allowPrivilegeEscalation=false, capabilities=Capabilities(add=[], drop=[ALL], additionalProperties={}), privileged=null, procMount=null, readOnlyRootFilesystem=true, runAsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, seccompProfile=null, windowsOptions=null, additionalProperties={}), startupProbe=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=null, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/glasskube/config, mountPropagation=null, name=config, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/glasskube/tls, mountPropagation=null, name=tls, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/vault/audit, mountPropagation=null, name=audit, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=ClusterFirst, enableServiceLinks=null, ephemeralContainers=[], hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostUsers=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, os=null, overhead={}, preemptionPolicy=null, priority=null, priorityClassName=null, readinessGates=[], resourceClaims=[], restartPolicy=Always, runtimeClassName=null, schedulerName=default-scheduler, schedulingGates=[], securityContext=PodSecurityContext(fsGroup=1000, fsGroupChangePolicy=null, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000, seLinuxOptions=null, seccompProfile=null, supplementalGroups=[], sysctls=[], windowsOptions=null, additionalProperties={}), serviceAccount=vault-test, serviceAccountName=vault-test, setHostnameAsFQDN=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=10, tolerations=[], topologySpreadConstraints=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=ConfigMapVolumeSource(defaultMode=420, items=[], name=vault-test, optional=null, additionalProperties={}), csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=config, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=tls, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=420, items=[], optional=null, secretName=vault-test-tls, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), additionalProperties={}), updateStrategy=StatefulSetUpdateStrategy(rollingUpdate=RollingUpdateStatefulSetStrategy(maxUnavailable=null, partition=0, additionalProperties={}), type=RollingUpdate, additionalProperties={}), volumeClaimTemplates=[PersistentVolumeClaim(apiVersion=v1, kind=PersistentVolumeClaim, metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={}, managedFields=[], name=audit, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PersistentVolumeClaimSpec(accessModes=[ReadWriteOnce], dataSource=null, dataSourceRef=null, resources=ResourceRequirements(claims=[], limits={}, requests={storage=1Gi}, additionalProperties={}), selector=null, storageClassName=null, volumeMode=Filesystem, volumeName=null, additionalProperties={}), status=PersistentVolumeClaimStatus(accessModes=[], allocatedResources={}, capacity={}, conditions=[], phase=Pending, resizeStatus=null, additionalProperties={}), additionalProperties={})], additionalProperties={}), status=StatefulSetStatus(availableReplicas=0, collisionCount=0, conditions=[], currentReplicas=1, currentRevision=vault-test-5bd5455664, observedGeneration=1, readyReplicas=null, replicas=1, updateRevision=vault-test-5bd5455664, updatedReplicas=1, additionalProperties={}), additionalProperties={}) 
 original desired: 
 {apiVersion=apps/v1, kind=StatefulSet, metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}, name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}]}, spec={persistentVolumeClaimRetentionPolicy={whenDeleted=Delete, whenScaled=Retain}, replicas=1, selector={matchLabels={app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https}, {containerPort=8201, name=https-internal}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={name=vault-test}, name=config}, {name=tls, secret={secretName=vault-test-tls}}]}}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}}}]}}  
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: metadata actual map value: {creationTimestamp=2023-08-25T09:00:45Z, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[{apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}, manager=vaultreconciler, operation=Apply, time=2023-08-25T09:00:45Z}, {apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}, manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z}], name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}], resourceVersion=2242, uid=33102e14-0069-4979-8739-64f86f00822e} managedFieldValue: {f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: labels actual map value: {app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0} managedFieldValue: {f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: spec actual map value: {podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector={matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https, protocol=TCP}, {containerPort=8201, name=https-internal, protocol=TCP}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], dnsPolicy=ClusterFirst, restartPolicy=Always, schedulerName=default-scheduler, securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccount=vault-test, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={defaultMode=420, name=vault-test}, name=config}, {name=tls, secret={defaultMode=420, secretName=vault-test-tls}}]}}, updateStrategy={rollingUpdate={partition=0}, type=RollingUpdate}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}, volumeMode=Filesystem}, status={phase=Pending}}]} managedFieldValue: {f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}} 
2023-08-25T11:01:36.566+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: persistentVolumeClaimRetentionPolicy actual map value: null managedFieldValue: {f:whenDeleted={}, f:whenScaled={}} 

Notice how in the last line, it says actual map value: null. This is not something keepOnlyManagedFields is designed to deal with, so that's where the error happens.

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

Thanks for your suggested workaround, we already removed the offending statement, since we want to support version 1.26 anyways. I just thought that the SDK could handle this edge-case more gracefully 🙂

Edit: For testing I created a cluster like this:

minikube start --profile=legacy --kubernetes-version 1.26

[shawkins]

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

I believe that case is happening as well here: kubernetes/kubernetes#118519 (comment)

on short notice you might want to try the generic matcher

While you won't get the NPE the matching won't work for pruned fields - the match logic will see that desired state has something that is missing in the actual.

[csviri]

see #2038