How to solve Promtail extraScrapeConfigs not being picked up? - kubernetes

Question

It seems that excluding logs in a pod using the configuration below does not work.
extrascrapeconfig.yaml:
- job_name: kubernetes-pods-app
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_label_name
###
- action: keep
regex: ambassador
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_namespace
###
To Reproduce
Steps to reproduce the behavior:
Deployed helm loki-stack :
helm install loki grafana/loki-stack --version "${HELM_CHART_VERSION}" \
--namespace=monitoring \
--create-namespace \
-f "loki-stack-values-v${HELM_CHART_VERSION}.yaml"
loki-stack-values-v2.4.1.yaml:
loki:
enabled: true
config:
promtail:
enabled: true
extraScrapeConfigs: extrascrapeconfig.yaml
fluent-bit:
enabled: false
grafana:
enabled: false
prometheus:
enabled: false
Attach grafana to loki datasource
Query: {namespace="kube-system"} in Grafana Loki
RESULT:
See logs
Expected behavior:
Not seeing any logs
Environment:
Infrastructure: Kubernetes
Deployment tool: Helm
What am I missing?

Answer 1

If you need Helm to pick up a specific file and pass it as a value, you should not pass the value itself in the values YAML file, but via another flag when installing or upgrading the release.
The command you are using is just applying the Helm values as-is, since the -f flag does not support parsing other files into the values by itself. Instead, use --set-file, which works similarly to --set, but gets the value content from the passed file.
Your command would now look like this:
helm install loki grafana/loki-stack --version "${HELM_CHART_VERSION}" \
--namespace=monitoring \
--create-namespace \
-f "loki-stack-values-v${HELM_CHART_VERSION}.yaml" \
--set-file promtail.extraScrapeConfigs=extrascrapeconfig.yaml