The podman stats command returns Error: Link not found #7883

xomachine · 2020-10-02T08:03:30Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
The podman stats command shows Error: Link not found for containers configured to use --network slirp4netns when running podman as root.

Steps to reproduce the issue:

Run podman container as root with --network slirp4netns option (E. g. podman run -d --network slirp4netns --name test --rm centos:7.5 /bin/sleep 600)
Ask podman to show stats via podman stats command for that container (E. g. podman stats test)

Describe the results you received:

The podman stats command returns Error: Link not found

Describe the results you expected:

The stats of the container to be shown in terminal as it happens when the stats command is run against the container without the --network slirp4netns option.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15
Built:        Mon Sep 28 05:07:17 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.18-1.el7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 892da24a64708c93d1cb1477074767b7348388cd'
  cpus: 20
  distribution:
    distribution: '"centos"'
    version: "7"
  eventLogger: journald
  hostname: ga-podman.test.local
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 3.10.0-1127.8.2.el7.x86_64
  linkmode: dynamic
  memFree: 97538551808
  memTotal: 270309453824
  ociRuntime:
    name: runc
    package: runc-1.0.0-15.2.el7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: 50d5773810ff80bea305931897920daa58a6a769
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 77824
  swapTotal: 4294963200
  uptime: 154h 34m 11.67s (Approximately 6.42 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 11
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,redirect_dir=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601258837
  BuiltTime: Mon Sep 28 05:07:17 2020
  GitCommit: ""
  GoVersion: go1.15
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman
podman-2.1.1-4.el7.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Podman run in the qemu virtual machine

The text was updated successfully, but these errors were encountered:

rhatdan · 2020-10-02T15:24:44Z

Not crazy about supporting this on RHEL7, Could you try this on a RHEL8 box and see if you have similar issues?

xomachine · 2020-10-02T17:02:51Z

Not quite RHEL8, but I also tested it on Archlinux and got the same behavior. Later will try to reproduce on RHEL8

# podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Git Commit:   9f6d6ba0b314d86521b66183c9ce48eaa2da1de2
Built:        Sat Sep 26 18:50:37 2020
OS/Arch:      linux/amd64

# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 35a2fa83022e56e18af7e6a865ba5d7165fa2a4a'
  cpus: 4
  distribution:
    distribution: '"antergos"'
    version: unknown
  eventLogger: journald
  hostname: xom-base
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.8.10-arch1-1
  linkmode: dynamic
  memFree: 5393444864
  memTotal: 15653163008
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc92
      commit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
      spec: 1.0.2-dev
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 20480909312
  swapTotal: 20480909312
  uptime: 12h 6m 48.09s (Approximately 0.50 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 1
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601135437
  BuiltTime: Sat Sep 26 18:50:37 2020
  GitCommit: 9f6d6ba0b314d86521b66183c9ce48eaa2da1de2
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

rhatdan · 2020-10-02T18:52:10Z

@AkihiroSuda @giuseppe Any idea what is going on here?

xomachine · 2020-10-06T07:01:32Z

Reproduced on CentOS 8 (both image and host)

# podman run --rm -d --network slirp4netns centos:8 sleep 60
af9b63d561f1fff310d6e55ebff3dc453d5911b79e51c12e931b62bc9f93b1b2
# podman stats af9b63d561f1fff310d6e55ebff3dc453d5911b79e51c12e931b62bc9f93b1b2
Error: Link not found

System info:

# podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.13.15
Built:        Fri Oct  2 17:30:39 2020
OS/Arch:      linux/amd64
# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: fa5f92225c4c95759d10846106c1ebd325966f91-dirty'
  cpus: 8
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: podman8.test.local
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  linkmode: dynamic
  memFree: 4283510784
  memTotal: 16643387392
  ociRuntime:
    name: runc
    package: runc-1.0.0-15.1.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: 2e01c9e4dfdab4b4c993a3698059fb824cd3286a
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2133323776
  swapTotal: 2147479552
  uptime: 3829h 17m 34.72s (Approximately 159.54 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 6
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601649039
  BuiltTime: Fri Oct  2 17:30:39 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

github-actions · 2020-11-06T00:13:37Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2020-11-23T12:40:25Z

Is this still happening?

@vrothberg This seems similar to the issue we have seen with podman stats in the CI/CD system? Do you think your PR would fix this?

vrothberg · 2020-11-24T08:16:29Z

@vrothberg This seems similar to the issue we have seen with podman stats in the CI/CD system? Do you think your PR would fix this?

It looks like another issue. The one I caused (and fixed) recently related to parsing cgroups. This one here seems to relate to networking.

xomachine · 2020-12-08T06:20:03Z

Still able to reproduce with podman 2.2.0

mheon · 2020-12-08T14:55:36Z

Found it. Our check for slirp4netns was insufficiently robust to catch the root + slirp case.

Just checking for `rootless.IsRootless()` does not catch all the cases where slirp4netns is in use - we actually allow it to be used as root as well. Fortify the conditional here so we don't fail in the root + slirp case. Fixes containers#7883 Signed-off-by: Matthew Heon <matthew.heon@pm.me>

mheon · 2020-12-08T15:00:59Z

Fix in #8648

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 2, 2020

github-actions bot added the stale-issue label Nov 6, 2020

rhatdan removed the stale-issue label Nov 23, 2020

mheon self-assigned this Dec 8, 2020

mheon added the In Progress This issue is actively being worked by the assignee, please do not work on this at this time. label Dec 8, 2020

mheon mentioned this issue Dec 8, 2020

Make podman stats slirp check more robust #8648

Merged

openshift-merge-robot closed this as completed in #8648 Dec 8, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The podman stats command returns Error: Link not found #7883

The podman stats command returns Error: Link not found #7883

xomachine commented Oct 2, 2020

rhatdan commented Oct 2, 2020

xomachine commented Oct 2, 2020

rhatdan commented Oct 2, 2020

xomachine commented Oct 6, 2020

github-actions bot commented Nov 6, 2020

rhatdan commented Nov 23, 2020

vrothberg commented Nov 24, 2020

xomachine commented Dec 8, 2020

mheon commented Dec 8, 2020

mheon commented Dec 8, 2020

The podman stats command returns Error: Link not found #7883

The podman stats command returns Error: Link not found #7883

Comments

xomachine commented Oct 2, 2020

rhatdan commented Oct 2, 2020

xomachine commented Oct 2, 2020

rhatdan commented Oct 2, 2020

xomachine commented Oct 6, 2020

github-actions bot commented Nov 6, 2020

rhatdan commented Nov 23, 2020

vrothberg commented Nov 24, 2020

xomachine commented Dec 8, 2020

mheon commented Dec 8, 2020

mheon commented Dec 8, 2020