Silverblue toolbox script broke because rootless 'podman create' can't chown $XDG_DATA_HOME/containers/storage/vfs

[debarshiray]

/kind bug

Description

Here is the toolbox script that we have been working on for Fedora Silverblue.

If you follow the README.md until the fedora-toolbox create step, then you'll see that the podman create ... command fails:

$ ./fedora-toolbox --verbose create
...
...
error creating libpod runtime: chown /var/home/rishi/.local/share/containers/storage/vfs: operation not permitted
./fedora-toolbox: failed to create container fedora-toolbox-rishi:28

This works fine if I use podman-0.9.1.1 with the fix for #1452 cherry-picked on top, but breaks if I use the 0.9.2 release.

Note that you'll need the patch from runc PR #1862 or you have to rollback to fedora-toolbox commit a878a1fe40e4c24a.

Output of podman version:

Version:       0.9.3-dev
Go Version:    go1.10.4
OS/Arch:       linux/amd64

Note that this is the podman-0.9.2-2.git81df604.fc28.x86_64 build which is the 0.9.2 release with the version number bumped post-release.

Output of podman info:

host:
  Conmon:
    package: podman-0.9.2-2.git81df604.fc28.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: d4d87a3d480359ecd97a2e53f418d8c6d2d68446-dirty'
  MemFree: 10861133824
  MemTotal: 16696311808
  OCIRuntime:
    package: runc-1.0.0-53.1.dev.git70ca035.fc28.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 4208979968
  SwapTotal: 4208979968
  arch: amd64
  cpus: 4
  hostname: bollard
  kernel: 4.18.7-200.fc28.x86_64
  os: linux
  uptime: 10m 23.68s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 1
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /var/home/rishi/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 3
  RunRoot: /run/user/1000/run

Additional environment details (AWS, VirtualBox, physical, etc.):

This is a physical laptop running Fedora 28 Silverblue 28.20180918.0, with podman and runc overlaid on top (see above).

[mheon]

I think we vendored some c/storage changes for 0.9.2, so I would imagine it's some of those

[giuseppe]

this should be fixed by: #1521

[giuseppe]

also, we need better tests for the rootless containers, we should not regress on such a basic feature. Not sure if we can get it working inside of a container but the hardest part is to setup multiple mappings for an user (via /etc/subuid), @cevich how could we setup another test running on F28?

[cevich]

@giuseppe good thinking. This sounds like something that would be in the territory of system-testing. Since those don't actually exist yet, for the near-term perhaps something light-weight could be added at the integration-test level (I'm guessing)?

Looking forward, @ypu is getting ginkgo-based system tests added by another PR. The goal is to have those tests consumed by external tooling, at and after packaging.

Reasoning: Since nothing gets exercised, w/o actually writing test content. I'm assuming it's easy to port a go-based integration test, over to a ginkgo system-test later on. That's why I suggest adding something at the integration-level, then we port/expand it later. Sound feasible?

[giuseppe]

@cevich we have some integration tests for the rootless case, but they are forced to use only one mapping as we don't have an user configured on the system. This is helpful but won't catch regressions like this one :(

We could probably take advantage of the ginkgo tests, we will still need to configure an user on the system that has multiple IDs available.

[cevich]

@giuseppe gotcha, ya most. def. the arena of system-testing then. I'm keen to get @ypu PR merged, I think it's 90% there and perhaps even good'nuff as-is. I'll keep my eyes on it.

[debarshiray]

Yes, this is fixed in podman-0.9.3 which contains the patch from PR #1521, (but it now breaks at a later stage due to #1535).