policy controller restarts if helm release name is not "policy-controller"

[prudnitskiy]

Description

If policy controller deployed via HELM and name of the release is not policy-controller - both policy-contoller and webhook pods continiously restarts. Looks like they can't create and populate CA bundle to configmap.

Log messages from webhook (excepmpt):

{"level":"info","ts":1678795519.842686,"logger":"fallback","caller":"webhook/main.go:94","msg":"Initializing TUF root from  => https://sigstore-tuf-root.storage.googleapis.com"}
main.go:102: {
  "gitVersion": "v0.7.0",
  "gitCommit": "89ef9041a0e3048fb50b54be2e288df9970736a0",
  "gitTreeState": "clean",
  "buildDate": "2023-02-27T19:26:41Z",
  "goVersion": "go1.19.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}
main.go:228: Registering 2 clients
main.go:229: Registering 2 informer factories
main.go:230: Registering 3 informers
main.go:231: Registering 3 controllers
{"level":"info","ts":"2023-03-14T12:05:20.624Z","logger":"policy-controller","caller":"profiling/server.go:64","msg":"Profiling enabled: false","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.631Z","logger":"policy-controller","caller":"leaderelection/context.go:47","msg":"Running with Standard leader election","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.646Z","logger":"policy-controller","caller":"sharedmain/main.go:283","msg":"Starting configuration manager...","commit":"89ef904-dirty"}
{"level":"info","ts":1678795520.7463984,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller","caller":"webhook/webhook.go:198","msg":"Informers have been synced, unblocking admission webhooks.","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller","caller":"sharedmain/main.go:311","msg":"Starting controllers...","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.defaultingwebhook.00-of-01 will run in leader-elected mode with id \"policy-controller-webhook-56bd45fc8b-g8glj_19b022af-1251-48f8-8f2b-942d59829786\"","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.DefaultingWebhook","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.DefaultingWebhook","caller":"controller/controller.go:496","msg":"Started workers","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.webhookcertificates.00-of-01 will run in leader-elected mode with id \"policy-controller-webhook-56bd45fc8b-g8glj_89e54dab-a864-4d00-b04f-71198cca9b18\"","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.WebhookCertificates","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.WebhookCertificates","caller":"controller/controller.go:496","msg":"Started workers","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.validationwebhook.00-of-01 will run in leader-elected mode with id \"policy-controller-webhook-56bd45fc8b-g8glj_776dc785-e002-4f06-8780-132b2c714dab\"","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.ValidationWebhook","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"89ef904-dirty"}
{"level":"info","ts":"2023-03-14T12:05:20.846Z","logger":"policy-controller.ValidationWebhook","caller":"controller/controller.go:496","msg":"Started workers","commit":"89ef904-dirty"}
I0314 12:05:20.846999       1 leaderelection.go:248] attempting to acquire leader lease security/policy-controller.defaultingwebhook.00-of-01...
I0314 12:05:20.847150       1 leaderelection.go:248] attempting to acquire leader lease security/policy-controller.webhookcertificates.00-of-01...
I0314 12:05:20.847230       1 leaderelection.go:248] attempting to acquire leader lease security/policy-controller.validationwebhook.00-of-01...
{"level":"warn","ts":"2023-03-14T12:05:39.791Z","logger":"policy-controller","caller":"webhook/webhook.go:154","msg":"server key missing","commit":"89ef904-dirty"}
server.go:3230: http: TLS handshake error from 10.0.0.1:46296: tls: no certificates configured
{"level":"warn","ts":"2023-03-14T12:05:40.791Z","logger":"policy-controller","caller":"webhook/webhook.go:154","msg":"server key missing","commit":"89ef904-dirty"}
server.go:3230: http: TLS handshake error from 10.0.0.1:37602: tls: no certificates configured

HELM chart never goes to the "ready" state in this situation.

Everything works as expected if the release name is exactly policy-controller

Version

• policy-controller: 0.7.0
• chart version: 0.5.2 -> 0.5.4
• kubernetes version: 1.24.9-gke.3200

[hectorj2f]

@prudnitskiy Your issue seems related to the leases. You don't need to specify a particular name for the helm release. The tls: no certificates configured comes from the webhooks not acquiring the leases. To do so, you just need to manually delete the leases kubectl delete leases -n cosign-system (cosign-system or any namespace you've chosen to deploy the policy-controller). Once the leases are deleted the webhook will be able to re-acquire them after a short while.

[hectorj2f]

@prudnitskiy Could you confirm that the deletion of the leases fixed your problem ?

[prudnitskiy]

@hectorj2f yes, lease cleanup solves the problem. Should I add it to the documentation, or is there a fix upcoming?

[hectorj2f]

@prudnitskiy Yes, please add it to the documentation. We could add a Troubleshooting.md to the docs.

[hectorj2f]

@prudnitskiy Fixed by sigstore/helm-charts#510! Closing this issue!