Cannot install splunk/splunk:latest on Ubuntu Server 18.04.1 and Docker available via snap install

[djsweet]

This is the same underlying bug as #96 but with 100% reproducibility as of this writing.

Canonical is still shipping Docker 18.06.1-ce in the default snap repository. I intend to notify them that their Docker is out of date, but they're now five months behind so it's safe to assume that their upgrade process will take a while.

From a brand new Ubuntu Server 18.04.1 installation, after the initial round of apt update and apt upgrade, and following these steps:

$ sudo snap install docker
$ sudo docker pull splunk/splunk:latest
$ sudo docker run -d -p 8000:8000 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=<Any Password>' --name splunk-server splunk/splunk:latest

Then the following output results:

$ sudo docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                     PORTS               NAMES
86e7ebef9874        splunk/splunk:latest   "/sbin/entrypoint.sh…"   6 seconds ago       Exited (2) 3 seconds ago                       splunk-server
$ sudo docker logs splunk-server

PLAY [Run default Splunk provisioning] *****************************************
Friday 18 January 2019  01:19:25 +0000 (0:00:00.053)       0:00:00.053 ********

TASK [Gathering Facts] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1547774365.39-234000989980791/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1547774365.39-234000989980791/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}
        to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

Friday 18 January 2019  01:19:25 +0000 (0:00:00.323)       0:00:00.376 ********
===============================================================================
Gathering Facts --------------------------------------------------------- 0.32s

Further pertinent information:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic
$ uname -a
Linux ubuntu-1804-docker-validation 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.4
 Git commit:        e68fc7a
 Built:             Mon Oct  1 14:25:31 2018
 OS/Arch:           linux/amd64
 Experimental:      false
$ snap info docker
name:      docker
summary:   Docker container runtime
publisher: canonical
contact:   snappy-devel@lists.ubuntu.com
license:   unknown
description: |
  Build and run container images with Docker.

  This build requires all files that Docker uses, such as dockerfiles, to be in $HOME. Keep files
  there for 'docker build', 'docker save' and 'docker load'. Use the 'docker-privilege' command for
  'docker run --privileged'.

  This snap is built by Canonical based on source code published by Docker, Inc. It is not endorsed
  or published by Docker, Inc.

  Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United
  States and/or other countries. Docker, Inc. and other parties may also have trademark rights in
  other terms used herein.
commands:
  - docker.compose
  - docker
  - docker.help
  - docker.machine
services:
  docker.dockerd: simple, enabled, active
snap-id:      sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ
tracking:     stable
refresh-date: today at 01:03 UTC
channels:
  stable:          18.06.1-ce   (321) 102MB -
  candidate:       18.06.1-ce   (321) 102MB -
  beta:            18.06.1-ce   (321) 102MB -
  edge:            17.03.1-ce-1 (333) 38MB  -
  17.09/stable:    –
  17.09/candidate: 17.09.1-ce   (232) 44MB  -
  17.09/beta:      17.09.1-ce   (232) 44MB  -
  17.09/edge:      17.09.1-ce   (232) 44MB  -
  17.06/stable:    17.06.2-ce   (179) 43MB  -
  17.06/candidate: 17.06.2-ce   (179) 43MB  -
  17.06/beta:      ↑
  17.06/edge:      17.06.2-ce-1 (237) 44MB  -
  17.03/stable:    17.03.2-ce-1 (159) 42MB  -
  17.03/candidate: 17.03.2-ce-1 (159) 42MB  -
  17.03/beta:      ↑
  17.03/edge:      17.03.2-ce-1 (159) 42MB  -
installed:         18.06.1-ce   (321) 102MB -

I can confirm that using the official Docker CE Ubuntu repository with Docker CE 18.09.1 results in a successful installation.

[nwang92]

I'm still unable to reproduce this, but I think I have a good feeling why it's happening.

I took a look into this some more, and I'm pretty sure it's caused by the introduction of acl library on the image-side. We need this to ensure access control of the Splunk filesystem, but part of the reason why it's failing for you is because by default, docker (or at least, an older version of docker) will default to using aufs which doesn't support acl. I believe at one point, docker did update CE to prefer overlay2 or overlay over the default aufs storage-driver.

If you still have an environment where this issue occurs, if you docker inspect <container-id> you will likely find the default graphdriver supporting the image filesystem is aufs. As a workaround, you can probably change the default storage-drive by including this json blob in your /etc/docker/daemon.json:

{
  "storage-driver": "overlay2"
}

You can likely get away with devicemapper as well, although having worked with Docker + swarm at scale, they have recommended to stick to overlay2.

Relevant link: https://groups.google.com/forum/#!topic/docker-user/165AARba2Bk

[djsweet]

That seems to be the fix, switching the default storage driver to overlay2. In Docker 18.06.1 the default is still aufs, and this seems like something that should be documented. I'll post a comment to the official documentation since that's outside of the scope of this project.

It's worth mentioning for anyone else who stumbles over this that the configuration location for the docker snap is at /var/snap/docker/current/config/daemon.json instead of /etc/docker/daemon.json.

[J-CMartin]

I'm using Splunk on a QNAP NAS and it also only works up to version 7.2.1. I run docker inspect and it seems to be using overlay2, but I still get the ansible error on the newest 7.2.4 image. Any other ideas?

"GraphDriver": { "Data": { "LowerDir": "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/412f5dd94453e3f7a692f8e05b1e238e35a713e9f5dce4b1e25ccc3f6f038d15-init/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/e5a5acc3fcadb6684c080f00dd91c5ced4fd9e55838e2887940716e981c58d32/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/d51c5e94d86aaae9bc7928f3f75e86206020f8974d9730d00b96b9732d27755e/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/95476209ec50e7ab2777d5c5a13425f7d02a9a05c2cf46435d8e4723cd340562/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/b58ffc6ff640a2e108b97b725963fb172535a3672614df8bc8eb02bc2e82d0a4/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/93e3e91db587012ddfa9db1a58e8643e6ceb80d3eb999e8fd4ef83546e386efc/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/97c76234084a4e2f27ea9193d96c751870618da3ad3b94b6fd7f0767c64fc301/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/49231e09f777698841dc0e0f72cf621dca2c9a70382cb99a4ac13b3af4794d59/diff:/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/1868af1cba399f5b4dd26fec710debb17b81c3bac6ce7a2749beeb9bfb92d0b0/diff", "MergedDir": "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/412f5dd94453e3f7a692f8e05b1e238e35a713e9f5dce4b1e25ccc3f6f038d15/merged", "UpperDir": "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/412f5dd94453e3f7a692f8e05b1e238e35a713e9f5dce4b1e25ccc3f6f038d15/diff", "WorkDir": "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/overlay2/412f5dd94453e3f7a692f8e05b1e238e35a713e9f5dce4b1e25ccc3f6f038d15/work" }, "Name": "overlay2"

Error msg:

TASK [Gathering Facts] ********************************************************* fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/t mp/ansible-tmp-1550423956.89-58475364311420/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1550423956.89-58475364311420/AnsiballZ_setup.py': Operation not permitted\ n}). For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"} to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

[nwang92]

@J-CMartin I don't think we've ever tested or run environments where the Docker root dir is backed entirely by NAS filesystem. My guess is this might be related to permissions between the host OS and the mount itself?

Additionally, I'm not too certain on the specifics or best practices around this, but you may hit performance issues with Splunk when indexing data if you run with NAS-backed storage (particularly with writing hot buckets as it can be IO intensive).

[diaznet]

Hello,

since change introduced in 7.2.3 (see comment in #96 ), we're unable to start up new version of container 7.2.5.
Driver is not aufs, but btrfs.

Any way to introduce a possibility to do like before 7.2.3 ?
Or support btrfs ?

[DeviantEng]

Like @diaznet, I'm running into the same thing but I'm specifically using the zfs driver for docker. 7.2.1 works, anything newer won't. :(