Splunk Universal Forwarder issue with 7.2.3 on Docker for Mac

[LorenKeagle]

As of 7.2.3 image from Dockerhub I am unable to map a forwarder app into the Splunk etc folder due to a permission problem:

splunkforwarder_1  |
splunkforwarder_1  | TASK [splunk_universal_forwarder : Install Splunk universal forwarder] *********
splunkforwarder_1  | fatal: [localhost]: FAILED! => {"changed": false, "dest": "/opt", "extract_results": {"cmd": ["/bin/tar", "--extract", "-C", "/opt", "-z", "--owner=splunk", "--group=splunk", "-f", "/var/tmp/ansible-tmp-1548625987.94-258334867164991/source"], "err": "/bin/tar: splunkforwarder/etc/apps/learned: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/learned: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/learned/metadata: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/learned: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/learned/metadata/default.meta: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/learned: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/learned/default: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/learned: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/learned/default/README: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/metadata: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/metadata/default.meta: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/default: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/default/transforms.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/default/restmap.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/default/app.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/search: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/search/default/props.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/metadata: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/metadata/default.meta: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/default-mode.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/web.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/app.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/server.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/README: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/default: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/default/app.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/default/README: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/bin: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/introspection_generator_addon/bin/collector.path: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/splunk_httpinput: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/splunk_httpinput: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/splunk_httpinput/default: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/apps/splunk_httpinput: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/apps: Cannot utime: Operation not permitted\n/bin/tar: Exiting with failure status due to previous errors\n", "out": "", "rc": 2}, "gid": 0, "group": "root", "handler": "TgzArchive", "mode": "0755", "msg": "failed to unpack /var/tmp/ansible-tmp-1548625987.94-258334867164991/source to /opt", "owner": "root", "size": 4096, "src": "/var/tmp/ansible-tmp-1548625987.94-258334867164991/source", "state": "directory", "uid": 0}
splunkforwarder_1  | 	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry
splunkforwarder_1  |
splunkforwarder_1  | PLAY RECAP *********************************************************************
splunkforwarder_1  | localhost                  : ok=1    changed=0    unreachable=0    failed=1
splunkforwarder_1  |
splunkforwarder_1  | Sunday 27 January 2019  21:53:11 +0000 (0:00:03.268)       0:00:04.903 ********
splunkforwarder_1  | ===============================================================================
splunkforwarder_1  | splunk_universal_forwarder : Install Splunk universal forwarder --------- 3.27s
splunkforwarder_1  | Gathering Facts --------------------------------------------------------- 1.26s
splunkforwarder_1  | Provision role ---------------------------------------------------------- 0.13s
splunkforwarder_1  | Upgrade role ------------------------------------------------------------ 0.05s
splunkforwarder_1  | Download pre-setup playbooks -------------------------------------------- 0.05s
splunkforwarder_1  | Run pre-setup playbooks ------------------------------------------------- 0.05s
splunkforwarder_1  | Download pre-setup playbooks -------------------------------------------- 0.04s
splunkforwarder_1  | ERROR: Couldn't read "/opt/splunkforwarder/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?
sa-demo_splunkforwarder_1 exited with code 2

As soon as I remove my app's volume mount from /opt/splunkforwarder/etc/apps everything starts up again. I presume this has to do with some permission changes made regarding the Ansible user. I suspect that my app volume mapping is creating the /opt/splunkforwarder/etc/apps directory as a different user before the ansible script is run, and ansible is unable to deal with the permission difference?

This was a very convenient method of 'installing' a forwarder app for local usage. I suppose I can build the app into my image, but I'd prefer to keep things running the way they were. Any recommendations on how to get permissions to work correctly with this scenario?

[lephino]

Create an ansible play as a "pre-task" that does a chown on /opt/splunkforward. It'll run before the install play and set the ownership back to "splunk:splunk".

[LorenKeagle]

My greatest fear was that the answer was going to involve "learn a new tool" LOL! :-)

1. How do I define an ansible pre-task? Is there a folder I can simply copy a script file in?
2. What user/group should I set in chown? ansible:ansible?

[lephino]

You can actually copy our entire task for this:

https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/change_splunk_directory_owner.yml

The thing to notice is, in ansible --- in the first line signals the start of the file, and then the yaml definition will define what module to run a command on. In this case, it's ansible's file module: https://docs.ansible.com/ansible/latest/modules/file_module.html#file-module

We're going the file module to become root, but then set the directory to the splunk user. No different then doing like a "sudo chown splunk:splunk /opt/splunkforwarder". The variables are just taking place for the user / directory.

Now to use it, there's a few ways to include it, you can map it to /tmp/splunk_ansible_pre_tasks.yml or you can create the file, and host it on a webserver and set an env var "ansible_pre_tasks". You can also set to just use our actual play by setting an ansible_pre_tasks to file:///opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml.

[LorenKeagle]

I've given your recommendations a shot, but I don't think they are having any positive effect. I'm not certain that the /tmp/splunk_ansible_pre_tasks.yml I created is actually being executed.

The very end of the error output still says the same thing:

Exiting with failure status due to previous errors\n", "out": "", "rc": 2}, "gid": 0, "group": "root", "handler": "TgzArchive", "mode": "0755", "msg": "failed to unpack /var/tmp/ansible-tmp-1548663281.65-264095580488254/source to /opt", "owner": "root", "size": 4096, "src": "/var/tmp/ansible-tmp-1548663281.65-264095580488254/source", "state": "directory", "uid": 0}

I'm not 100% sure how to interpret this, but I believe it's still saying that the /opt directory is still owned by root. Therefore, the /tmp/splunk_ansible_pre_tasks.yml isn't enough on its own. Is there something else I need to do to get it to execute?

Out of curiosity, I've updated both the splunk and splunkforwarder images to 7.2.3, and I'm not having any issues with the splunk image. I'm able to map my app on my host maching using a volume directly into /opt/splunk/etc/apps. Is there some reason that the containers should behave differently?

[nwang92]

@LorenKeagle there's actually an initial task that runs that should change permissions of everything under /opt/splunk to the splunk user. See https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/change_splunk_directory_owner.yml

Also regarding your latest comment, did upgrading to 7.2.3 resolve this issue for you? Maybe I'm a bit confused, I thought the initial problem you had was with the 7.2.3 image. Nevermind, turns out I can't read. It seems like the problem is with the universalforwarder image entirely, and not the splunk image. I think I know the problem if that's the case, but can you confirm?

[LorenKeagle]

@nwang92 That's correct!

Some background. I initially reported a startup issue for splunk forwarder in #32. Once I saw that was fixed and in Dockerhub, I updated my Dockerfile to pull from 7.2.3 to try it out, and then ran into this issue.

The splunk image works great using volumes to map apps from my host into /opt/splunk/etc/apps, but this fails in the splunk forwarder due to some permissions issue. Perhaps the ansible scripts between the two images are different? I would think the installation steps would be very similar between the two, but there's apparently a discrepancy.

If there is an official supported way to 'preload' a local app into the container, please let me know. Maybe there's a better way to go about this that utilizes your existing ansible setup tasks.

[nwang92]

Got it, yes I'm seeing the difference between the behavior of splunk/splunk and splunk/universalforwarder. I can get that fixed as part of the upcoming 7.2.4 release.

The latest Splunk image (7.2.3) does have limited app support, basically anything from Splunkbase or any http://... link. There was an issue with support https:// links, but that's resolved within the develop branch only right now.

You can create a container with something like docker run -d -e SPLUNK_PASSWORD=helloworld -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_APPS_URL=http://webserver/app.tgz splunk/universalforwarder to automatically install an app bundle. I'm open to more feedback on how people use their apps + Splunk environments, but I imagine this would be useful amongst those running Splunk in docker because:

1. It follows the model of "declarative" infrastructure - if you need a Splunk deployment with a particular app or set of apps, you should be able to define that at boot-time
2. There is some flexibility in where apps come from - maybe certain teams like using Splunkbase, maybe others would prefer to host/cache/build independent apps internally. Either way, there should be a little flexibility in terms of "bringing-your-own-infrastructure" to support your stack.

But again, completely open to feedback or suggestions on this. I think your use case is valid and we should support that going forward. The only "issue" of bind-mounting apps is it may not be a very portable solution when transitioning to running these Splunk containers in a large cluster.

[LorenKeagle]

@nwang92 Awesome that you found a difference! I was hoping I wasn't crazy :-D

In our case, our app is not public, as it potentially exposes IP and product capabilities to our competitors. However, my use case for this environment is intended to be a local, completely isolated demo/POV setup, with all required components already available in the repository. The app is already part of the local repo, so it would not be desired to have to upload it to an HTTP server just so the build script can access it. If there's a way to reference a local archive or mapped folder (either via a volume or through a COPY build command) that would be ideal.

[nwang92]

Going to close this - the code is currently in develop right now if you want to build your own images for now. But it will be released as part of 7.2.4 (expected to come out next week). I also added a test case that bind-mounts full app directories to the running container and validates that apps get registered in Splunk with this PR: #120