Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Get information that system rolled back #118

Open
pmtk opened this issue Sep 24, 2023 · 3 comments
Open

Get information that system rolled back #118

pmtk opened this issue Sep 24, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@pmtk
Copy link

pmtk commented Sep 24, 2023

Hello

In MicroShift we're looking for a sure way to know that system rolled back so we can perform certain actions.

We have some ideas so far:

  1. In red script, if boot_counter == 0 then create some kind of file on disk to persist that information and read it on next boot. This assumes that:
    • Existence of boot_counter means we're in the middle of "new deployment hasn't been determined yet to be okay, so greenboot might reboot the system" - especially when red script runs (as it will be followed by reboot)
    • boot_counter == 0 means it's a last attempt, when system reboots, grub will see that value and select second boot entry (rollback)
  2. Inspect journalctl --boot 0 -u greenboot-rpm-ostree-grub2-check-fallback for existence of FALLBACK BOOT DETECTED! Default rpm-ostree deployment has been rolled back message
    • Slightly worried about timing, we would probably want After=greenboot-rpm-ostree-grub2-check-fallback, but we'd have to check if non-ostree has any impacts, or if RemainAfterExit=yes would affect that as well
  3. Extend greenboot so greenboot-rpm-ostree-grub2-check-fallback to create a file like /run/rolled-back, this file would be removed by greenboot-grub2-set-counter or be cleaned automatically by reboot (if new deployment wasn't staged, but machine was simply rebooted)

Do you have any other ideas how could we make it a robust mechanism?
Would you happen to know about any other source of this information like grub or (rpm-)ostree?

@say-paul
Copy link
Member

say-paul commented Sep 26, 2023

Does the for a sure way to know that system rolled back needs to be known on the boot(T) when greenboot will attempt rollback or on reboot(T+1) after the rollback happened. It will determine which approach needs to be taken.
Though I agree there should be a cleaner way to determine it.

@pmtk
Copy link
Author

pmtk commented Sep 26, 2023

The latter: on reboot(T+1) after the rollback happened

@say-paul
Copy link
Member

Then the possible workaround for now can be to check if the the boot_counter is unset/boot_success=1 combined with journalctl --boot 1 -u greenboot-rpm-ostree-grub2-check-fallback has the rollback message.

@say-paul say-paul added the enhancement New feature or request label Sep 27, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants