Separate filebeat daemon set based on namespace? - kubernetes

Question

I am trying to run the filebeat daemon set to get the log for particular app. There are basically two nodegroups:- eai and eai-staging. eai nodgroup have only single namespace by the eai-staging have multiple namespace.
I have following filebeat config:
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: log
fields:
app_type: "${NAMESPACE}". <<<---- I want this app type to be different based on namespace
log_type: secure
fields_under_root: true
output.logstash:
hosts: ["${LOGSTASH_HOST}:${LOGSTASH_PORT}"]
ttl: 1s
pipelining: 0
processors:
- drop_fields:
fields: ["beat", "host", "input", "offset", "source"]
Filebeat Daemon set
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
labels:
app: filebeat
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: filebeat
spec:
nodeSelector:
nodegroup: eai
priorityClassName: critical
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
imagePullPolicy: Always
image: docker.elastic.co/beats/filebeat:6.5.4
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: LOGSTASH_HOST
value: "logstash-headless.etl.svc.cluster.local"
- name: LOGSTASH_PORT
value: "5046"
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: app-log
mountPath: /var/log/app/
readOnly: true
volumes:
- name: config
configMap:
name: filebeat-config
- name: data
hostPath:
path: /var/lib/filebeat-data/eai/app-filebeat
type: DirectoryOrCreate
- name: app-log
hostPath:
path: /var/log/app/
type: DirectoryOrCreate
Now, how can I get the particular namespace from where the app log is obtained by the filebeat.
I tried to deploy one daemon set in the eai namespace in the eai nodegroup. So I can get the namespace for that using metadata.namespace.
But, if I deployed the daemon set in the eai-staging node group in the particular namespace. I will always get the same namespace value.
Is there any way around. Or should I deploy the daemon set in each namespace?
P.S. I could not use the filebeat in the same container because if filebeat is down due to some reason, the pod cannot receive the request for the app

Answer 1

Deploy filebeat as daemonset in each node and filebeat will get logs from all containers in that node but you can add namespace, pod name, labels as metadata to each event. This way you will get to know from which namespace the event was originated.
The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes pod the event originated from. Each event is annotated with:
Pod Name
Namespace
Labels
https://www.elastic.co/guide/en/beats/filebeat/6.1/add-kubernetes-metadata.html