Podman doesn't use a local image in `FROM` in Dockerfile/Containerfile

[AndesKrrrrrrrrrrr]

Issue Description

When building devcontainers, the FROM in the dynamic containerfile that is used references a newly tagged image, but the FROM step prompts me for which registry to choose (which should've been localhost).

When re-running the image name doesn't change, but I'm still prompted, so does not seem to be a race condition.

Steps to reproduce the issue

Steps to reproduce the issue

1. git clone https://github.com/island-is/island.is && cd island.is
2. devcontainer up --workspace-folder .
3. Get prompted

Describe the results you received

Get promted.

Describe the results you expected

No prompt, and build continues

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 95.5
    systemPercent: 1.37
    userPercent: 3.13
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: silverblue
    version: "40"
  eventLogger: journald
  freeLocks: 1944
  hostname: thonk
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.11-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1557872640
  memTotal: 33295925248
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 7483551744
  swapTotal: 8589930496
  uptime: 48h 25m 9.00s (Approximately 2.00 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/person/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 2
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/person/.local/share/containers/storage
  graphRootAllocated: 1022488477696
  graphRootUsed: 714292277248
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1315
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/person/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.0
  Built: 1716940800
  BuiltTime: Wed May 29 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Running in a toolbox, but wrapping podman and docker with a flatpak-spawn --host script. It has never caused issues.

Additional information

No response

[baude]

can you replicate this outside devcontainers? like with a containerfile ?

[rhatdan]

This looks like you did not fully specify the image name, just used a short name?

[AndesKrrrrrrrrrrr]

@baude, I haven't tried, the issue seems to be short name resolution anyways.

@rhatdan, I did not specify anything, rather it's the devcontainer cli pulling from a locally built layer.

Is it wrong of me to assume that we should search locally first for short-names? I would've expected this if the image being pulled is of a conventional registry form (e.g. foo/bar)... I guess it's all a repositories, slashes or not. Nonetheless, I would expect local-first search before asking for for registry selection.

Also, why isn't localhost/vsc-cli-* an option? I can copy the string and run locally with

podman run --rm -it vsc-cli-*

I could post an issue to the devcontainer cli asking to prefix the image pull with localhost/... 🤔

[AndesKrrrrrrrrrrr]

Here's some sample output during devcontainer build in the https://github.com/devcontainers/cli repo:

podman build -f /tmp/devcontainercli-person/updateUID.Dockerfile-0.65.0 -t vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006-uid --platform linux/amd64 --build-arg BASE_IMAGE=vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006 --build-arg REMOTE_USER=node --build-arg NEW_UID=1000 --build-arg NEW_GID=1000 --build-arg IMAGE_USER=node /tmp/devcontainercli-person/empty-folder
STEP 1/9: FROM vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006
? Please select an image: 
  ▸ registry.fedoraproject.org/vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006:latest
    registry.access.redhat.com/vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006:latest
    docker.io/library/vsc-cli-4ee3098167f24595df8161e2a6a226ab0bfe78567231ed22d21f0f0d45a5d006:latest

[rhatdan]

@vrothberg PTAL

[vrothberg]

git clone -b feat/local-dev https://github.com/island-is/island.is

The branch doesn't exist anymore. I assume it has been merged?

[vrothberg]

Here's some sample output during devcontainer build in the https://github.com/devcontainers/cli repo:

Works on my end. Did you configure /etc/containers/registries.conf? For instance, the short-name-mode option?

[vrothberg]

Note: requires installing the podman-docker package as it seems that devcontainer shells out to the client rather than using the socket.

[AndesKrrrrrrrrrrr]

The branch doesn't exist anymore

Yes, merged to main. I updated the description.

Did you configure /etc/containers/registries.conf?

Hmm... It seems like, I did configure my registry.conf:

# # An array of host[:port] registries to try when pulling an unqualified image, in order.
# unqualified-search-registries = ["example.com"]
unqualified-search-registries = ["registry.fedoraproject.org", "registry.access.redhat.com", "docker.io"]

Nonetheless, I'm would still expect localhost to be searched for short-name image pulls.

[AndesKrrrrrrrrrrr]

I added localhost to the unqualified-search-registries, and seems to have fixed the issue.

Again, I would like for localhost to be searched even if it weren't in the list

[rhatdan]

I agree that should be the default behaviour, and not doing this is a bug.

podman build -t foobar
podman run foobar

Should work.

[AndesKrrrrrrrrrrr]

Should this also work?

podman build -t foobar

cat <<EOF >> Containerfile.dynamic
FROM foobar
RUN cowsay
EOF

podman build -f Containerfile.dynamic

I would expect so, but apparently it doesn't. This is what devcontainers does; builds an image, then uses it in a dynamically generated containerfile, which is what is causing this issue.

[rhatdan]

$ cat /tmp/Containerfile
from alpine
run echo hi
$ podman build -t foobar /tmp
STEP 1/2: FROM alpine
STEP 2/2: run echo hi
--> Using cache f2222d280005d435f8debfcfdf9809893feeb0bad4f35504801017de87714622
COMMIT foobar
--> f2222d280005
Successfully tagged localhost/foobar:latest
f2222d280005d435f8debfcfdf9809893feeb0bad4f35504801017de87714622
 $ podman images | grep foobar
localhost/foobar          latest      f2222d280005  2 minutes ago       8.09 MB
$ cat /tmp/Containerfile.dynamic 
from foobar
run echo hi again
$ podman build -f /tmp/Containerfile.dynamic -t foobar1 /tmp
STEP 1/2: FROM foobar
STEP 2/2: run echo hi again
hi again
COMMIT foobar1
--> 68f7823fd0dc
Successfully tagged localhost/foobar1:latest
68f7823fd0dc5eae28725e908ffbc8fde549762490221b2469e59f48535dece7
podman images | grep foobar
localhost/foobar1         latest      68f7823fd0dc  3 minutes ago  8.09 MB
localhost/foobar          latest      f2222d280005  4 minutes ago  8.09 MB

[rlanting]

A reproducer I found because my podman-compose was sometimes working and sometimes it wasn't:

$ cat Containerfile                      
FROM docker.io/library/alpine
RUN echo "hello"
$ podman build -t test-container  .
$ podman run -it --rm test-container echo "This runs"        
This runs
$ podman run --platform="linux/amd64" -it --rm test-container echo "This runs"
? Please select an image: 
  ▸ registry.fedoraproject.org/test-container:latest
    registry.access.redhat.com/test-container:latest
    docker.io/library/test-container:latest

--platform is the issue here

Running with debug logs gets me this:

DEBU[0000] Enforcing pull policy to "newer" to pull custom platform (arch: "amd64", os: "linux", variant: "") - local image may mistakenly specify wrong platform

which originates from https://github.com/containers/common/blob/ae4a61e1b2e0af84a668f87f7622d86ebc418cba/libimage/pull.go#L560 and that links to #10682

Looking in the git blame for the if statement, it looks like there was an attempt to fix it but it broke something else.

And then there is short-name-mode = "enforcing" in the default registries.conf which makes it ask when interactive, and fail when non-interactive.

All of these things together gets you into this state. To be fair picking any of the registries (or even ctrl+c) does start the container if you're running interactively even though it doesn't exist in the remote registry. But that does leak local container names.

Reproduced on podman 5.1.2 but 4.9.x also has this issue.