Kubernetes promtail sidecar: how to get labels from the parent pod metadata - kubernetes

Question

I have some kubernetes applications that log to files rather than stdout/stderr, and I collect them with Promtail sidecars. But since the sidecars execute with "localhost" target, I don't have a kubernetes_sd_config that will apply pod metadata to labels for me. So I'm stuck statically declaring my labels.
# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: promtail
name: sidecar-promtail
data:
config.yml: |
client:
url: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
backoff_config:
max_period: 5m
max_retries: 10
min_period: 500ms
batchsize: 1048576
batchwait: 1s
external_labels: {}
timeout: 10s
positions:
filename: /tmp/promtail-positions.yaml
server:
http_listen_port: 3101
target_config:
sync_period: 10s
scrape_configs:
- job_name: sidecar-logs
static_configs:
- targets:
- localhost
labels:
job: sidecar-logs
__path__: "/sidecar-logs/*.log"
----
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-logger
spec:
selector:
matchLabels:
run: test-logger
template:
metadata:
labels:
run: test-logger
spec:
volumes:
- name: nfs
persistentVolumeClaim:
claimName: nfs-claim
- name: promtail-config
configMap:
name: sidecar-promtail
containers:
- name: sidecar-promtail
image: grafana/promtail:2.1.0
volumeMounts:
- name: nfs
mountPath: /sidecar-logs
- mountPath: /etc/promtail
name: promtail-config
- name: simple-logger
image: foo/simple-logger
volumeMounts:
- name: nfs
mountPath: /logs
What is the best way to label the collected logs based on the parent pod's metadata?

Answer 1

You can do the following:
In the sidecar container, expose the pod name, node name and other information you need as environment variables, then add the flag '-config.expand-env' to enable environment expansion inside promtail config file, e.g.:
...
- name: sidecar-promtail
image: grafana/promtail:2.1.0
# image: grafana/promtail:2.4.1 # use this one if environment expansion is not available in 2.1.0
args:
# Enable environment expansion in promtail config file
- '-config.expand-env'
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
...
Then in your configMap, add the environment variables in your static_config labels as such:
...
scrape_configs:
- job_name: sidecar-logs
static_configs:
- targets:
- localhost
labels:
job: sidecar-logs
pod: ${POD_NAME}
node_name: ${NODE_NAME}
__path__: "/sidecar-logs/*.log"
...