Datadog: API Key invalid dropping transaction when installing Datadog agent - kubernetes

Question

I'm trying to install Datadog agent for a Kubernetes cluster using Helm.
This is the helm command I'm using for it:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install datadog datadog/datadog \
--namespace monitoring \
--create-namespace \
--atomic \
--set datadog.apiKey=<MY-DATADOG-API-KEY> \
--set targetSystem=linux \
--values values.yaml
Values file:
datadog:
kubelet:
host:
valueFrom:
fieldRef:
fieldPath: spec.nodeName
hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
tlsVerify: false # Required as of Agent 7.35. See Notes.
However, when I run the liveness probe error with error 500 which shows the error below:
CLUSTER | ERROR | (pkg/forwarder/transaction/transaction.go:344 in internalProcess) | API Key invalid, dropping transaction for https://orchestrator.datadoghq.com/api/v1/orchestrator.

Answer 1

Here's how I solved it:
The issue had to do with the Datadog Destination Site. The Destination site for my metrics, traces, and logs is supposed to be datadoghq.eu. This is set using the variable DD_SITE, and it defaults to datadoghq.com if it is not set.
To check what your Datadog Destination Site just look at the URL of your Datadog dashboard:
For US it will be - https://app.datadoghq.com/
For EU it will be - https://app.datadoghq.eu/
To set this in your helm chart simply do either of the following:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install datadog datadog/datadog \
--namespace monitoring \
--create-namespace \
--atomic \
--set datadog.apiKey=<MY-DATADOG-API-KEY> \
--set targetSystem=linux \
--set datadog.site=datadoghq.eu \
--values values.yaml
OR set it in your values file:
datadog:
site: datadoghq.eu
kubelet:
host:
valueFrom:
fieldRef:
fieldPath: spec.nodeName
hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
tlsVerify: false # Required as of Agent 7.35. See Notes.
References:
Datadog Agent Forwarder fails liveness probe when new spot instance joins cluster, causing multiple restarts #1697
DD_SITE Set to us3.datadoghq.com, but process-agent and security-agent Still Try to Connect to non us3 endpoints #9180