Investigate unreliability of docker GitHub Action in mantle

[krnowak]

Description

Recently the "docker" GitHub Action in mantle became very flaky. The failure is during installation of the python packages with apt-get:

#15 402.3 Setting up python3-wrapt (1.14.1-2+b2) ...
#15 403.3 Traceback (most recent call last):
#15 403.3   File "/usr/bin/py3compile", line 323, in <module>
#15 403.3     main()
#15 403.3   File "/usr/bin/py3compile", line 302, in main
#15 403.3     compile(files, versions,
#15 403.3   File "/usr/bin/py3compile", line 203, in compile
#15 403.3     interpreter.magic_number(version),
#15 403.3     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#15 403.3   File "/usr/share/python3/debpython/interpreter.py", line 233, in magic_number
#15 403.3     result = self._execute('import imp; print(imp.get_magic())', version)
#15 403.3              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#15 403.3   File "/usr/share/python3/debpython/interpreter.py", line 359, in _execute
#15 403.3     raise Exception('{} failed with status code {}'.format(command, output['returncode']))
#15 403.3 Exception: ('python3.11', '-c', 'import imp; print(imp.get_magic())') failed with status code -11
#15 403.4 dpkg: error processing package python3-wrapt (--configure):
#15 403.4  installed python3-wrapt package post-installation script subprocess returned error exit status 1

The above snippet means that calling python3.11 -c 'import imp; print(imp.get_magic())' resulted in SIGSEGV (signal 11) when configuring some python package.

What we have tried:

• bumped the buildx action to v6 - Update github action version to v6 and go to 1.23 mantle#575
• then reverting it - Revert ".github: Update docker/build-push-action to v6" mantle#576
• splitting the installation of packages into two steps (install python3 first, then install the rest of packages) - hack mantle#577
• patching /usr/share/python3/debpython/interpreter.py to avoid using the deprecated imp module - hack mantle#577
• downgrading the debian image from bookworm to bullseye - Try debian bullseye instead of bookworm mantle#578

What seems to be a working workaround is running apt-get install in a loop - flatcar/mantle#579

But it's obviously ugly.

Impact

Now, with the workaround, the action mostly works, but we don't want to keep having such solutions. Without it, CI tests were failing for arm64 builds because the mantle image doesn't exist, because the action for arm64 package failed.

Environment and steps to reproduce

1. Revert Dockerfile: Repeat apt-get several times until it succeeds mantle#579
2. Kick off the "docker" GH Action