Installation failure for grub-efi-amd64-signed on ubuntu-latest

[sushantmimani]

Description

sudo apt-get -y upgrade --fix-missing fails with the following error on ubuntu-latest

mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
[122](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:123)
dpkg: error processing package grub-efi-amd64-signed (--configure):
[123](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:124)
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
[124](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:125)
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
[125](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:126)
Processing triggers for hicolor-icon-theme (0.17-2) ...
[126](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:127)
Processing triggers for man-db (2.10.2-1) ...
[127](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:128)
Errors were encountered while processing:
[128](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:129)
 grub-efi-amd64-signed
[129](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:130)
needrestart is being skipped since dpkg has failed
[130](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:131)
E: Sub-process /usr/bin/dpkg returned an error code (1)

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: ubuntu-22.04
Version: 20230219.1
https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774

Is it regression?

No

Expected behavior

Command should execute successfully.

Actual behavior

Command is failing.

I've downgraded to ubuntu-20.04 where this command runs successfully.

Repro steps

1. Create a workflow that uses ubuntu-latest
2. Run command sudo apt-get -y upgrade --fix-missing

[christophebedard]

Is it regression?

No

I would argue it is a regression.

[ddobranic]

@sushantmimani hello, thank you for reporting. until the issue is fixed, please use this workaround:

echo RESET grub-efi/install_devices | debconf-communicate grub-pc

[mikhailkoliada]

Hello!

echo 'APT::Get::Always-Include-Phased-Updates "false";' | sudo tee /etc/apt/apt.conf.d/99-phased-updates

Is your friend. We have phased updates enabled on Ubuntu, this usually helps new updates to be delivered faster, but sometimes it also breaks some compatibility / upgrades (if someone runs ones during canonical updates). You may wish to add it to your workflows if you are running updates often on your own. Feel free to ask more questions if you have, I am going to close the ticket as there is nothing actable for us right now.

[tothxa]

@mikhailkoliada Are you sure it's a problem with phased upgrades?

The relevant part of the failure log is:

Setting up grub-efi-amd64-signed (1.187.3~22.04.1+2.06-2ubuntu14.1) ...
mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

As far as I can tell, phased upgrades should result in packages being held back, but here grub is getting upgraded (even though I don't think many of the affected workflows will actually ever use the upgraded version). edit: sorry, my mistake, you said to disable phased upgrades, not enable them, but that doesn't change the part that this is a local config error

A changed UUID is a local problem instead, isn't it?

[congyiwu]

Adding notes here b/c I hit this in my dev environment while fixing a different image for Larger Runners.

This smells a lot like canonical/cloud-init#4091, which is currently still open. GitHub Actions runners still boot with BIOS. If I'm correct, this issue could reoccur whenever the grub-efi-amd64-signed package is updated twice within a short period of time, if:

1. grub-efi-amd64-signed is updated in the package repo
2. We build the Ubuntu image for GitHub Actions with a base image that doesn't have the new grub-efi-amd64-signed version.
3. The grub-efi/install_devices debconf key is initially unset
4. During the image build, we'll upgrade grub-efi-amd64-signed:

   runner-images/images/linux/scripts/base/repos.sh

   Line 16 in 9d1c613

   apt-get -yq dist-upgrade

   
   The logs will probably look like this:

   Trying to migrate /boot/efi into esp config
   Installing grub to /boot/efi.
   Installing for x86_64-efi platform.
   grub-install: warning: EFI variables cannot be set on this system.
   grub-install: warning: You will have to complete the GRUB setup manually.
   Installation finished. No error reported.
   

   This also sets grub-efi/install_devices to a path in /dev/disk/by-id/ based on the current ID of the disk
5. We release the image
6. grub-efi-amd64-signed is updated in the package repo again
7. We create a runner with the image.
   This resets the disk ID
8. A customer runs apt-get upgrade
   They'll get an error like this:

   Setting up grub-efi-amd64-signed (1.187.3~22.04.1+2.06-2ubuntu14.1) ...
   mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
   dpkg: error processing package grub-efi-amd64-signed (--configure):
    installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
   

And it looks this reoccurred last week: https://github.com/orgs/community/discussions/47863#discussioncomment-7212869

From looking at Jammy/updates in https://launchpad.net/ubuntu/+source/grub2-signed/+publishinghistory", the timing sort of lines up:
* 1.187.2 was published on 2023-01-30
* 1.187.3~22.04.1 was published on 2023-02-16
* 1.187.4~22.04.1 was published on 2023-09-28
* 1.187.6 was published on 2023-10-04