policy-controller: Uninstalling the chart does not clean up leases

[eddiezane]

Description

There is an issue that occurs when you install the policy controller chart, uninstall it, and then install it again.

The problem is the leases are not being cleaned up from the namespace causing the reinstalled pods to fail to start.

# pods/policy-controller-policy-webhook

I0410 22:46:49.210762       1 leaderelection.go:248] attempting to acquire leader lease cosign-system/clusterimagepolicy.defaultingwebhook.00-of-01...

# pods/policy-controller-policy-webhook

{"level":"warn","ts":"2023-04-10T22:47:15.135Z","logger":"policy-controller","caller":"webhook/webhook.go:154","msg":"server key missing","commit":"89ef904-dirty"}
server.go:3230: http: TLS handshake error from 10.244.0.1:52812: tls: no certificates configured
{"level":"warn","ts":"2023-04-10T22:47:16.135Z","logger":"policy-controller","caller":"webhook/webhook.go:154","msg":"server key missing","commit":"89ef904-dirty"}
server.go:3230: http: TLS handshake error from 10.244.0.1:52824: tls: no certificates configured

# this is after uninstall

 kubectl -n cosign-system get leases.coordination.k8s.io
NAME                                                                                                             HOLDER                                                                                   AGE
clusterimagepolicy.conversionwebhook.00-of-01                                                                    policy-controller-policy-webhook-5985d96f75-szpt6_39f3f639-fe73-4f35-bccb-22ee126c441a   5m40s
clusterimagepolicy.defaultingwebhook.00-of-01                                                                    policy-controller-policy-webhook-5985d96f75-szpt6_71ff5327-f0b9-4142-a0d9-cc22aa7da344   5m40s
clusterimagepolicy.github.heygears.com.sigstore.policy-controller.pkg.reconciler.clusterimagepolicy.reconciler.00-of-01   policy-controller-policy-webhook-5985d96f75-szpt6_6ab347b6-ad24-4c36-b80d-353c96664c99   5m40s
clusterimagepolicy.github.heygears.com.sigstore.policy-controller.pkg.reconciler.trustroot.reconciler.00-of-01            policy-controller-policy-webhook-5985d96f75-szpt6_f250d67a-8d6b-43b9-a0f0-768c31abc825   5m40s
clusterimagepolicy.validationwebhook.00-of-01                                                                    policy-controller-policy-webhook-5985d96f75-szpt6_6c977a6d-b0dc-415b-a9dc-47e0eb9979d9   5m40s
clusterimagepolicy.webhookcertificates.00-of-01                                                                  policy-controller-policy-webhook-5985d96f75-szpt6_1131c65e-3c5e-4272-921f-7a13a176723d   5m40s
policy-controller.defaultingwebhook.00-of-01                                                                     policy-controller-webhook-69dc7d6b4b-g4mxv_0b708261-c3b7-4971-a2eb-550fc38f9b34          5m39s
policy-controller.validationwebhook.00-of-01                                                                     policy-controller-webhook-69dc7d6b4b-g4mxv_f8603637-0216-450d-838e-a62657b3ecbf          5m39s
policy-controller.webhookcertificates.00-of-01                                                                   policy-controller-webhook-69dc7d6b4b-g4mxv_4b032a04-a0ff-4fdd-a778-6b53c85a2bfd          5m39s

Helm no longer manages namespaces for users so the fix is to instruct the user to delete the namespace (not ideal) or to use some sort of uninstall hook to clean up the leases. I think the latter is what we want.

Steps to reproduce

1. kind create cluster
2. kubectl create ns cosign-system
3. helm install -n cosign-system policy-controller sigstore/policy-controller
4. wait for healthy pods...
5. helm uninstall -n cosign-system policy-controller
6. wait for pods to be removed...
7. helm install -n cosign-system policy-controller sigstore/policy-controller
8. Check the pod logs and see errors

Version

Chart version: 0.5.1
App version: 0.7.0

[hectorj2f]

Thanks for the issue @eddiezane, these leases are created by the knative services on start. For the moment, we recommend to manually delete the leases. But I agree we need to add an uninstall hook.

[vaikas]

Can we add instructions for manually doing these if we can't do the hooks in a near future?

[hectorj2f]

Let's fix it and avoid extra documentation: #510