Update 4.16.0-0.okd-scos-2024-09-24-151747 stuck. APIServicesAvailable: PreconditionNotReady

[alexminder]

Describe the bug
OKD update stuck from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-09-24-151747

$ oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.0-0.okd-scos-2024-09-24-151747   False       False         False      4d21h   APIServicesAvailable: PreconditionNotReady
baremetal                                  4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
cloud-controller-manager                   4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
cloud-credential                           4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
cluster-autoscaler                         4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
config-operator                            4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
console                                    4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      9d
control-plane-machine-set                  4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
csi-snapshot-controller                    4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      52d
dns                                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      51d
etcd                                       4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      7d1h
image-registry                             4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      52d
ingress                                    4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      4d
insights                                   4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
kube-apiserver                             4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
kube-controller-manager                    4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
kube-scheduler                             4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
kube-storage-version-migrator              4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      17d
machine-api                                4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
machine-approver                           4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
machine-config                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      3d
marketplace                                4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
monitoring                                 4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      3d
network                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      503d
node-tuning                                4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      3d23h
openshift-apiserver                        4.16.0-0.okd-scos-2024-09-24-151747   False       False         False      4d21h   APIServicesAvailable: PreconditionNotReady
openshift-controller-manager               4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      52d
openshift-samples                          4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      3d23h
operator-lifecycle-manager                 4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
operator-lifecycle-manager-catalog         4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
operator-lifecycle-manager-packageserver   4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      3d1h
service-ca                                 4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d
storage                                    4.16.0-0.okd-scos-2024-09-24-151747   True        False         False      503d

$ oc adm upgrade
info: An upgrade is in progress. Unable to apply 4.16.0-0.okd-scos-2024-09-24-151747: some cluster operators are not available

Upgradeable=False

  Reason: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade
  Message: Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on nodes hw-testing2-okd-01.tutu.ru, hw-testing2-okd-02.tutu.ru, and hw-testing2-okd-03.tutu.ru will not be supported in the next OpenShift minor version upgrade.

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-scos-4
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.

openshift-apiserver-operator log have:

connectivity_check_controller.go:169] ConnectivityCheckController is waiting for transition to desired version (4.16.0-0.okd-scos-2024-09-24-151747) to be completed.

Version
4.15.0-0.okd-scos-2024-01-18-223523

How reproducible
oc adm upgrade --force --allow-explicit-upgrade --to-image=quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-09-24-151747

Log bundle
must-gather

[JaimeMagiera]

Hi,

We’ve not tested any upgrades from the older SCOS builds from January, nor do we have the ability to troubleshoot those. They were nightlies, which generally aren’t necessarily expected to interact with updates. The forced upgrade process is meant as a way to get from official 4.15 FCOS releases -> 4.16 SCOS — all of which is still just in testing. There is now official 4.16 SCOS release yet. We’ll have an update on the project in the next few days.

Thanks.

[alexminder]

@JaimeMagiera

They were nightlies

The release was in stable channel. How can I understand where nightly release and where prod ready?

[JOECHONG0420]

@alexminder I have fixed this issue. Please create the configmap below can fix the error. This error appear when the first time installation, the bootstrap node not automatically remove. So it will missing the configmap in kube-system namespace.

kind: ConfigMap
apiVersion: v1
metadata:
name: bootstrap
namespace: kube-system
data:
status: complete

After create this configmap, the installation will process.
I can upgrade from FCOS 4.15.0-0.okd-2024-03-10-010116 -> SCOS 4.16.0-okd-scos.1
Hope you can upgrade success.

[alexminder]

@alexminder I have fixed this issue. Please create the configmap below can fix the error. This error appear when the first time installation, the bootstrap node not automatically remove. So it will missing the configmap in kube-system namespace.

kind: ConfigMap apiVersion: v1 metadata: name: bootstrap namespace: kube-system data: status: complete

After create this configmap, the installation will process. I can upgrade from FCOS 4.15.0-0.okd-2024-03-10-010116 -> SCOS 4.16.0-okd-scos.1 Hope you can upgrade success.

@JOECHONG0420 , Thank you a lot! This helped me. How did you find it?

[JOECHONG0420]

@alexminder I have fixed this issue. Please create the configmap below can fix the error. This error appear when the first time installation, the bootstrap node not automatically remove. So it will missing the configmap in kube-system namespace.
kind: ConfigMap apiVersion: v1 metadata: name: bootstrap namespace: kube-system data: status: complete
After create this configmap, the installation will process. I can upgrade from FCOS 4.15.0-0.okd-2024-03-10-010116 -> SCOS 4.16.0-okd-scos.1 Hope you can upgrade success.

@JOECHONG0420 , Thank you a lot! This helped me. How did you find it?

I have many okd clusters, I compare the different between the cluster that can upgrade and the cluster that cannot upgrade. Then try to make two cluster to be the same. I find this configmap can make the upgrade process.