unable to map inputs.conf to latest forwarder container

[arungn-tii]

I used to be able to do this with splunkforwarder-7.2.1-be11b2c46e23-Linux (splunk/universalforwarder@sha256:cf9b53ee182de36671de774ffecd63f3cbf3ed7cb353a50804f14151e25d12be)

docker run -it -p 9777:9777 -v /etc/admin/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_INDEXER_URL=indexers.xxxx.com' -e 'SPLUNK_PASSWORD=xyz' splunk/universalforwarder:latest

Now with the latest, splunkforwarder-7.2.3-06d57c595b80-Linux (splunk/universalforwarder@sha256:9c99896c7fbfbb6e94689e33ca2cb621e3a49512b2322d7273be7c90774f8a42) it's throwing a permission error ansible is not able to unpack splunk files under /opt/splunkforwarder/etc. I noticed that the default user has changed from "splunk" to "ansible". What is the right way to map overrides in the container now ?

fatal: [localhost]: FAILED! => {"changed": false, "dest": "/opt", "extract_results": {"cmd": ["/bin/tar", "--extract", "-C", "/opt", "-z", "--owner=splunk", "--group=splunk", "-f", "/var/tmp/ansible-tmp-1548084479.3-222115422914359/source
"], "err": "/bin/tar: splunkforwarder/etc/log-cmdline.cfg: Cannot open: Permission denied\n/bin/tar: splunkforwarder/etc/log.cfg: Cannot open: Permission denied\n/bin/tar: splunkforwarder/etc/copyright.txt: Cannot open: Permission denied\
n/bin/tar: splunkforwarder/etc/init.d: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/init.d: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/init.d/README: Cannot open: No such file or directory\n/bin/tar:
splunkforwarder/etc/licenses: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/licenses: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/licenses/forwarder: Cannot mkdir: No such file or directory\n/bin/tar: s
plunkforwarder/etc/licenses: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/licenses/forwarder/splunkforwarder.lic: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission de
nied\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/b
in/tar: splunkforwarder/etc/modules/input/UDP: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/UDP/config.xml: Cannot open: No su
ch file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/structuredparsing: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: C
annot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/structuredparsing/config.xml: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforw
arder/etc/modules/input/fschangemanager: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/fschangemanager/config.xml: Cannot open:
 No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/TCP: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mk
dir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/TCP/config.xml: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/inpu
t/exec: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/exec/config.xml: Cannot open: No such file or directory\n/bin/tar: splunk
forwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkforwarder/etc/modules/input/FIFO: Cannot mkdir: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir: Permission denied\n/bin/tar: splunkfor
warder/etc/modules/input/FIFO/config.xml: Cannot open: No such file or directory\n/bin/tar: splunkforwarder/etc/modules: Cannot mkdir:  [snip]```

[lephino]

What are you placing in inputs.conf that you need to volume map it?

To your question, the ansible user has sudo rights, you can volume mount this in just as you're doing, and then run a ansible-pre-task to chown the directory. You could also use the {configs} vars option that's been added to have this file get generated every time with your settings applied instead of volume mounting.

[arungn-tii]

My inputs.conf only has a bunch of log paths to monitor.

host=hostname

[monitor:///var/log/nodejs/nodejs.log]
index=application
sourcetype=nodejs

[monitor:///var/log/app/nginx-access.log]
index=web-access
sourcetype=nginx_access

[monitor:///var/log/app/nginx-error.log]
index=application
sourcetype=nginx_error

[monitor:///var/log/app/fpm-error.log]
index=application
sourcetype=fpm_error

I forgot to add the log volume to the docker command earlier. The correct command is

docker run -it -p 9777:9777 -v /etc/admin/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf -v /var/log/app:/var/log/app -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_INDEXER_URL=indexers.xxxx.com' -e 'SPLUNK_PASSWORD=xyz' splunk/universalforwarder:latest

I'm new to splunk so what other methods can be used to add those log paths to forwarder? Or what's the right way to tell forwarder container to monitor some log paths on the host ?

[nwang92]

Just an update/FYI, I'm pretty sure this issue + #115 are caused by the same thing. There seems to be a missing line in the provisioning of the universalforwarder to update permissions for $SPLUNK_HOME.

[arungn-tii]

Thank you for confirming there's a bug. Will there be a new release out soon with a fix?

[nwang92]

@arungn-tii we typically release with Splunk maintenance release cadences, but as of now, we are planning to push a new image (7.2.4) with the fix sometime next week.

[nwang92]

Going to close this - the code is currently in develop right now if you want to build your own images for now. But it will be released as part of 7.2.4 (expected to come out next week). I also added a test case that bind-mounts full app directories to the running container and validates that apps get registered in Splunk with this PR: #120

[arungn-tii]

@nwang92 Hey. I see that a new version is out. But I am still not able to get it working. Could you tell me what I'm doing wrong? Should I use ansible or splunk cli to add inputs instead?

/home/admin# docker run -it  -p 9777:9777  -v /var/log/app:/var/log/app -v ~admin/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_INDEXER_URL=indexers.xxx.yyy.xom' -e 'SPLUNK_PASSWORD=testpass'   splunk/universalforwarder:7.2.4

PLAY [Run default Splunk provisioning] *******************************************************************************************************************************************************************************************************
Tuesday 19 February 2019  08:39:38 +0000 (0:00:00.116)       0:00:00.116 ******

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chown: changing ownership of '/var/tmp/ansible-tmp-1550565578.84-214227964818
640/': Operation not permitted\nchown: changing ownership of '/var/tmp/ansible-tmp-1550565578.84-214227964818640/AnsiballZ_setup.py': Operation not permitted\n}). For information on working around this, see https://docs.ansible.com/ansibl
e/become.html#becoming-an-unprivileged-user"}
        to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

Tuesday 19 February 2019  08:39:39 +0000 (0:00:00.862)       0:00:00.978 ******
===============================================================================
Gathering Facts ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.86s```

[nwang92]

@arungn-tii can you run a docker version and docker info for me and show me the output?

Additionally, can you confirm you're using overlay2 as the storage-driver? I would see the notes + comments about overlay2 here: #105

[arungn-tii]

I was running the container on a debian box which was using "auf" storage driver. On redhat systems, it seems to work fine with "devicemapper". I think we can close now. Thanks for the help.