Unable to install SNO on baremetal: issue with bootkube.service and release-image-pivot.service

[soubinan]

Hello,

I am trying to deploy the stable version 4.16.0-okd-scos.0 on baremetal
The installation flavour desired here is a Single Node Okd
But it fails due to an issue on release-image-pivot.service and bootkube.service (seems to be a consequence of the first, but not sure)

Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.360249182 +0000 UTC m=+0.062435394 container create f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release=4.16.0-okd-scos.0, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620)
Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.345887004 +0000 UTC m=+0.048073216 image pull 30779e283f1edc25841af6412a0dd2487e118018ea1589dd9d1bf5928d893bd7 quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de
Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.461770507 +0000 UTC m=+0.163956767 container init f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620, io.openshift.release=4.16.0-okd-scos.0)
Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.468989391 +0000 UTC m=+0.171175648 container start f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release=4.16.0-okd-scos.0, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620)
Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.469892453 +0000 UTC m=+0.172078713 container attach f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release=4.16.0-okd-scos.0, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620)
Nov 17 17:03:39 localhost.localdomain adoring_saha[18936]: quay.io/okd/scos-content@sha256:3079a2dfbec9a5ad7b2b7db8bdfeba666492a8ed046cd8bd8087666d1f6fc64b
Nov 17 17:03:39 localhost.localdomain podman[18918]: 2024-11-17 17:03:39.517370098 +0000 UTC m=+0.219556356 container died f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release=4.16.0-okd-scos.0, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620)
Nov 17 17:03:39 localhost.localdomain podman[18960]: 2024-11-17 17:03:39.612895999 +0000 UTC m=+0.079818397 container remove f64056b0da91801267500ecaff78cfaa9854a6454d9361fb86800ccf8afa0123 (image=quay.io/okd/scos-release@sha256:35242882450436957021e5b4ebdc40b8c8b21d09c6f922519bc3e958a5fa58de, name=adoring_saha, io.openshift.release=4.16.0-okd-scos.0, io.openshift.release.base-image-digest=sha256:ef19bf74a768ad7e8078abed5c1f62272ffc3f90e93203c19d7783286531e620)
Nov 17 17:03:39 localhost.localdomain bootkube.sh[18983]: /usr/local/bin/bootkube.sh: line 81: oc: command not found
Nov 17 17:03:39 localhost.localdomain systemd[1]: bootkube.service: Main process exited, code=exited, status=127/n/a
Nov 17 17:03:39 localhost.localdomain systemd[1]: bootkube.service: Failed with result 'exit-code'.
Nov 17 17:03:39 localhost.localdomain systemd[1]: bootkube.service: Consumed 6.295s CPU time.
Nov 17 17:03:44 localhost.localdomain systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 11.
Nov 17 17:03:47 localhost.localdomain systemd[1]: Started bootkube.service - Bootstrap a Kubernetes cluster.

/usr/local/bin/bootkube.sh: line 81: oc: command not found ...

░░ The unit var-lib-containers-storage-overlay.mount has successfully entered the 'dead' state.
Nov 17 17:38:19 localhost.localdomain rpm-ostree[4756]: client(id:cli dbus:1.336 unit:release-image-pivot.service uid:0) added; new total=1
Nov 17 17:38:19 localhost.localdomain rpm-ostree[4756]: Loaded sysroot
Nov 17 17:38:19 localhost.localdomain bootstrap-pivot.sh[16534]: error: Remounting /sysroot read-write: Permission denied
Nov 17 17:38:19 localhost.localdomain rpm-ostree[4756]: client(id:cli dbus:1.336 unit:release-image-pivot.service uid:0) vanished; remaining=0
Nov 17 17:38:19 localhost.localdomain rpm-ostree[4756]: In idle state; will auto-exit in 62 seconds
Nov 17 17:38:19 localhost.localdomain systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit release-image-pivot.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Nov 17 17:38:19 localhost.localdomain systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit release-image-pivot.service has entered the 'failed' state with result 'exit-code'.
Nov 17 17:38:19 localhost.localdomain systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
░░ Subject: A start job for unit release-image-pivot.service has failed

error: Remounting /sysroot read-write: Permission denied ...

[BeardOverflow]

See:

• 4.16.0-okd-scos.1 Sinlge node installation not working properly #27
• Pivots to SCOS fail due to newer ext4 features enabled in FCOS okd#2041

[soubinan]

Closed since there are already in progress discussions on it