debian/11/cloud doesn't run cloud-init in lxd

[antoonhuiskens]

Hi,
I tend to run images with lxd on non-lxd managed networks (unmanaged networks in lxd speak).

With Debian/11/cloud (bullseye/cloud) I run into the situation that networking doesn't come up, which seems to be related to the fact that cloud-init doesn't run, when compared to e.g. debian/12/cloud (bookworm/cloud).

Here's my environment:

$ snap list lxd
Name  Version       Rev    Tracking       Publisher   Notes
lxd   5.15-002fa0f  25112  latest/stable  canonical✓  -
$ lxc --version
5.15
$ lxc network list -f compact | egrep 'NAME|br100'
    NAME       TYPE    MANAGED  IPV4  IPV6  DESCRIPTION  USED BY  STATE
  br100      bridge    NO                                16
$ lxc remote ls -fcompact | egrep 'NAME|linux'
       NAME                          URL                       PROTOCOL      AUTH TYPE   PUBLIC  STATIC  GLOBAL
  images           https://images.linuxcontainers.org        simplestreams  none         YES     NO      NO
$ lxc profile show default  # output is edited for brevity
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br100
    type: nic
  root:
    path: /
   ...
name: default
used_by:
- /1.0/instances/d11c
- /1.0/instances/d12c

Note that at this stage I'm not passing in any cloud-init yet. (that happens in images that derive from this).

Now when I launch "vanilla" cloud images from both, I observe (in addition to networking being down) that cloud-init is disabled on bullseye but not on bookworm:

$ lxc launch --ephemeral images:debian/11/cloud d11c
Creating d11c
Starting d11c
$ lxc launch --ephemeral images:debian/12/cloud d12c
Creating d12c
Starting d12c
$ lxc exec d11c -- cloud-init status
status: disabled
$ lxc exec d12c -- cloud-init status
status: done

This behaviour is triggered by the systemd generator for cloud-init:

$ lxc exec d11c -- cat /run/cloud-init/cloud-init-generator.log
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (container[lxc]: pid 1 cmdline): /sbin/init
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]

whereas the debian/12/cloud image reports this:

$ lxc exec d12c -- cat /run/cloud-init/cloud-init-generator.log
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (container[lxc]: pid 1 cmdline): /sbin/init
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=0

Similarly:

$ for i in d11c d12c ; do echo -n "${i}: " ;lxc exec $i -- cat /run/cloud-init/.ds-identify.result ; done
d11c: 1
d12c: 0

In short: I think my woes are due to the fact that cloud-init is disabled due to a failing cloud-init datasource identification in bullseye, that works in bookworm.

Upon further investigation:

$ for i in d11c d12c ; do echo -n "${i}: " ;lxc exec $i -- fgrep '_LXD(' /usr/lib/cloud-init/ds-identify; echo ;done
d11c:
d12c: dscheck_LXD() {

And really, there's no lxd datasource capability defined:

$ for i in d11c d12c ; do echo -n "${i}: " ;lxc exec $i -- grep -A3 '^DI_DSLIST_DEFAULT' /usr/lib/cloud-init/ds-identify; echo ;done
d11c: DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud"
DI_DSLIST=""

d12c: DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack \
OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud VMware \
LXD NWCS"

My current take is that this lack of lxd capability is due to the cloud-init version:

$ for i in d11c d12c ; do echo "${i}: " ;lxc exec $i -- dpkg -l cloud-init ; echo ;done
d11c:
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version          Architecture Description
+++-==============-================-============-========================================================
ii  cloud-init     20.4.1-2+deb11u1 all          initialization system for infrastructure cloud instances

d12c:
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-========================================================
ii  cloud-init     22.4.2-1     all          initialization system for infrastructure cloud instances

When I install cloud-init from bullseye-backports and reboot (in order to trigger the systemd generator for cloud-init),
it all starts to work as expected and consistent with bookworm:

root@d11c:~# dpkg -l cloud-init
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Architecture Description
+++-==============-==============-============-========================================================
ii  cloud-init     22.2-1~bpo11+1 all          initialization system for infrastructure cloud instances
root@d11c:~# cloud-init status
status: done

Summed up, I'd like to make the case for setting up debian/11/cloud with cloud-init 22.2 from bullseye-backports since that version includes support for LXD environments (i.e. cloud-init is broken for bullseye) and will have consistent cloud-init behaviour across os families.

To the very least I hope the above documents the work around well enough, though implementing the workaround requires a reboot of the container in my packer build pipeline which is a bit of a burden.

[antoonhuiskens]

Below snippet is my from my packer workaround:

  provisioner "file" {
    destination = "/etc/apt/sources.list.d/bullseye-backports.list"

    content = <<-EOF
      deb http://deb.debian.org/debian bullseye-backports main
      EOF
    only = ["lxd.debian-bullseye-cloud"]
  }

  provisioner "shell" {
    inline = [
      "dhclient",
      "apt update && apt install -y -t bullseye-backports cloud-init",
      "systemctl enable systemd-networkd.service",
      "shutdown -r now"
    ]
    only        = ["lxd.debian-bullseye-cloud"]
    pause_after = "5s"

[stgraber]

Is this still happening? We have automated daily tests of our images and they've not flagged this image as having an issue.

[kkremitzki]

This may be caused by the debian/11/cloud image having an incorrect sources.list; unless one installs cloud-init from bullseye-backports, the file /etc/cloud/templates/sources.list.debian.tmpl will not have logic to handle the /updates -> -updates change.

root@d11-cloud:~# grep updates /etc/apt/sources.list
## Major bug fix updates produced after the final release of the
deb http://security.debian.org/ bullseye/updates main
deb-src http://security.debian.org/ bullseye/updates main
deb http://deb.debian.org/debian bullseye-updates main
deb-src http://deb.debian.org/debian bullseye-updates main
root@d11-cloud:~# apt update
Hit:1 http://deb.debian.org/debian bullseye InRelease
Ign:2 http://security.debian.org bullseye/updates InRelease
Hit:3 http://deb.debian.org/debian bullseye-updates InRelease
Err:4 http://security.debian.org bullseye/updates Release
  404  Not Found [IP: 199.232.30.132 80]
Hit:5 http://deb.debian.org/debian bullseye-backports InRelease
Reading package lists... Done
E: The repository 'http://security.debian.org bullseye/updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.