add_kubernetes_metadata needs to respect logs_path setting

[TomonoriSoejima]

Suppose you have this setting in filebeat.yml.

    processors:
      #- decode_json_fields:
      #    fields: message
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/pods/"
                resource_type: 'pod'

Then you have mounted container logs in the given path /var/log/pods/, but add_kubernetes_metadata can not find the match.

2021-10-21T16:41:48.474Z DEBUG [kubernetes] add_kubernetes_metadata/matchers.go:88 Incoming log.file.path value: /var/log/pods/openshift-xxxx-logging_mdsd-xxxxxxx/fluentbit/0.log 2021-10-21T16:41:48.474Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:210 No container match string, not adding kubernetes data {"libbeat.processor": "add_kubernetes_metadata"}

Because the code expects the path to be the static value /var/lib/kubelet/pods/

We should use the passed value in logs_path in the code to be more flexible.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[MichaelKatsoulis]

The podLogsPath is set in https://github.com/elastic/beats/blob/v7.8.0/filebeat/processor/add_kubernetes_metadata/matchers.go#L144

Currently only /var/lib/kubelet/pods is supported. This is not well documented.
We should support different log paths too.
We try at that point to extract the podUid.

Under /var/lib/kubelet/pods directory there is a subdirectory for every pod named by each pod's uid.

root@kind-control-plane:/var/lib/kubelet/pods# ls
26626464-11fe-4e3e-9584-787e89bbbb90  488f397b-4fd2-4af8-9783-35be8db5c050  783871022e445e7daf092834474bf987      ab7d4ace-0570-49e9-9182-0bca019d07cc  ffb4f461-bba7-4d20-ae98-c51dd709070c
33cba1e9-2f8c-4e70-b2bf-bcbd100662fe

Under /var/log/pods there is a subdirectory for each pod but the name of each one of them is constructed from the namespace_pod-name_pod-uid.

root@kind-control-plane:/var/log/pods# ls
default_redis_8467167a-9975-4cda-a409-633f7fce34c0                              kube-system_kube-controller-manager-kind-control-plane_46dac9a538838115821dfd9559149484
kube-system_coredns-558bd4d5db-4k8cb_ceca5ee4-77a6-48d8-ab41-8906cb4df7be       kube-system_kube-proxy-2lckw_a91bbf99-1a6f-443f-a1f6-2cb8f6a35ee6

So we need a way to extract the pod_uid from different log paths. But the constraint is that we need to know for each one of them where we can get the pod uid from the directory name.

@ChrsMark @tetianakravchenko

[tetianakravchenko]

After discussing this issue with @MichaelKatsoulis and @ChrsMark , sum up:

1. in current implementation we expect that the logs file and the file we extract pod/container id (from the file name) is the same - it is not true for pod now: logs are taken from /var/logs and pod ids from /var/lib/kubelet/pods/.
2. logs_path is misleading in case of resource_type: pod -> can be used hard-coded value here
   TODO: update documentation.
3. we don't want to mount /var/lib/kubelet/pods/ into the filebeat pod -> check how to extract the pod_uid from log path /var/log/pods ( /var/log path is already mounted).