Vault Injector with externalVaultAddr permission denied - kubernetes

Question

I'm trying to connect Vault injector from my AWS Kubernetes cluster and login using the injector.externalVaultAddr="http://my-aws-instance-ip:8200"
Installation:
helm install vault \
--set='injector.externalVaultAddr=http://my-aws-instance-ip:8200' \
/path/to/my/vault-helm
After following the necessary steps here: https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/ and customising this:
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat kube_token_from_my_k8s_cluster)" \
kubernetes_host="$KUBERNETES_HOST:443" \
kubernetes_ca_cert=#downloaded_ca_crt_from_my_k8s_cluster.crt
Now, after adding the deployment app.yaml and adding the vault annotations to read the secret/helloworld, it doesn't start my pod and shows this error on the container vault-agent-init
NAME READY STATUS RESTARTS AGE
app-aaaaaaa-bbbb 1/1 Running 0 34m
app-aaaaaaa-cccc 0/2 Init:0/1 0 34m
vault-agent-injector-xxxxxx-zzzzz 1/1 Running 0 35m
$ kubectl logs -f app-aaaaaaa-cccc -c vault-agent-init
...
URL: PUT http://my-aws-instance-ip:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* permission denied" backoff=2.769289902
I have also tried manually doing it in my local:
$ export VAULT_ADDR="http://my-aws-instance-ip:8200"
$ curl --request POST \
--data "{\"role\": \"myapp\", \"jwt\": \"$(cat kube_token_from_my_k8s_cluster)\"}" \
$VAULT_ADDR/v1/auth/kubernetes/login
# RESPONSE OUTPUT
{"errors":["permission denied"]}

Answer 1

Try curling the Kubernetes API-service with the values you used to set up Kubernetes auth in Vault.
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT"
curl --cacert <ca-cert-file> -H "Authorization: Bearer $TOKEN_REVIEW_JWT" $KUBE_HOST
If that doesn't work Vault won't be able to communicate with the cluster to get tokens verified.
I had this problem when trying to integrate Vault with a Rancher managed cluster, $KUBE_HOST was pointing to the rancher proxy so I needed to change it to the IP of the cluster and extract token and ca cert from the service account I was using.