Filebeat with custom logfile and kubernetes metadata to elasticsearch and kibana - kubernetes

Question

Our applications are deployed in AWS EKS cluster, and for certain reasons we need to write our app logs to separate file lets say ${POD_NAME}.applog instead of stdout (we mounted /var/log/container/ to the pod /log folder and app writes /log/${POD_NAME}.applog ). And we are using filebeat to send the logs to Elasticsearch and we are using Kibana for visualization. Our filebeat config file looks like this
data:
filebeat.yml: |-
filebeat.inputs:
- type: log
paths:
- /var/log/containers/*.applog
json.keys_under_root: true
json.message_key: log
processors:
- add_cloud_metadata:
- add_host_metadata:
This is working fine, but we realised we are missing the kuberenetes metadata in ES and Kibana. But we are getting kuberenetes metadata when we include -type: conatainer.
data:
filebeat.yml: |-
filebeat.inputs:
- type: log
paths:
- /var/log/containers/*.applog
json.keys_under_root: true
json.message_key: log
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
So we tried adding the config like this
data:
filebeat.yml: |-
filebeat.inputs:
- type: log
paths:
- /var/log/containers/*.applog
json.keys_under_root: true
json.message_key: log
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
- add_cloud_metadata:
- add_host_metadata:
Still we are not getting the kuberenetes metadata in kibana. I tried with all trial and error method, but nothing works.
Can someone please help me how to get Kubernetes metadata with custom logfile in filebeat.

Answer 1

We meet the same issue under EKS 1.24, there is no kubernetes metadata in the log entry. Per doc Add Kubernetes metadataedit, we solve it through the following config under filebeat version 7.17.8.
- type: container
paths:
- /var/log/pods/*/*/*.log
exclude_files: ['filebeat.*',
'logstash.*',
'kube.*',
'cert-manager.*']
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: '/var/log/pods/'
default_indexers.enabled: false
default_matchers.enabled: false
indexers:
- pod_uid:
matchers:
- logs_path:
logs_path: '/var/log/pods/'
resource_type: 'pod'
The default indexer and matcher should be disabled. And the pod_uid indexers could identify the pod metadata using the UID of the pod. You could find the logs_path matchers config from here