metrics-server errors

[noambenm]

Environmental Info:
K3s Version:
k3s version v1.31.6+k3s1 (6ab750f)
go version go1.22.12

Node(s) CPU architecture, OS, and Version:
Linux mdch-lab 6.12.17-v8-16k+ #1861 SMP PREEMPT Wed Mar 5 16:04:19 GMT 2025 aarch64 GNU/Linux

Cluster Configuration:
single-node cluster on Raspberry Pi 5

Describe the bug:
On several clean Installs of linux in different environments the metrics-server logs report errors regarding the readiness probe

kubectl get pod -n kube-system metrics-server-XXXXXXX :

1 server.go:191] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

kubectl describe pod -n kube-system metrics-server-XXXXX:

Warning Unhealthy 7m24s kubelet Readiness probe failed: Get "https://10.42.0.54:10250/readyz": dial tcp 10.42.0.54:10250: connect: connection refused
Warning Unhealthy 7m9s (x9 over 7m22s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500

The environments I have checked are Ubuntu on a vm (x86), Debian (arm) in aws ec2 instance and on my raspberry pi 5 running Pi OS.
In all cases linux is freshly installed and the installation of k3s is via the default script (I've tried with and without some flags)

Im not sure it it's related but when I deployed kube-prometheus-stack, I got similar errors from pods with the connection refused.

I did try to research the issue and found that increasing the initialDelaySecond from 0 to 20 or 15 seconds in the metrics-server deployment help, but Im not sure It will solve everything

Steps To Reproduce:
Just install k3s on any machine and check the logs and describe of the metrics-server pod

Additional context / logs:
journalctl -u k3s
filter relevant errors

Mar 07 18:02:16 mdch-lab k3s[761]: E0307 18:02:16.026295 761 handler_proxy.go:143] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
Mar 07 18:02:17 mdch-lab k3s[761]: E0307 18:02:17.017462 761 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService"
Mar 07 18:02:19 mdch-lab k3s[761]: E0307 18:02:19.643587 761 resource_quota_controller.go:175] "Unhandled Error" err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"
Mar 07 18:45:08 mdch-lab k3s[743]: E0307 18:45:08.093862 743 namespace_controller.go:164] "Unhandled Error" err="deletion of namespace cilium-secrets failed: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"
Mar 07 18:25:15 mdch-lab k3s[724]: Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

I would like to know if you have input how how to solve this issue, thank you very much!

[brandond]

This is normal during startup. Metrics-server only scrapes metrics from nodes periodically, and nodes take a while to start up and publish metrics to be scraped. It is normal for there to be no metrics served by metrics-server for a minute or so after the cluster starts. Just be patient.

If metrics are not available over a longer time period, make sure that the correct ports are open between nodes. The node hosting the metrics-server pod needs to be able to connect to the kubelet port on all other nodes. This is covered in the docs: https://docs.k3s.io/installation/requirements#networking

[noambenm]

Your explanation makes way more sense.😃
Thanks for the reply! @brandond