I'm new to using Kubernetes (v 1.11.2-gke.18), and am trying to get Sentry running as a Kubernetes deployment. This is all being run in Google Cloud. Here's the YAML files for Postgres, Redis, and Sentry:
The Redis and Postgres parts seem to be up and running just fine, but Sentry fails because: could not translate host name "postgres-sentry:5432" to address: Name or service not known.
My question is this: How do I get these services to communicate properly? How do I get the Sentry container to communicate with Postgres and Redis?
It turns out that in this case, what I needed was to change the SENTRY_REDIS_HOST and SENTRY_POSTGRES_HOST values to not have the port numbers in the env declaration. Taking those out made everything talk together as expected.
In Kubernetes pods generally can find other pods through services using DNS.
Your configuration looks correct if postgres-sentry is in the same namespace as your sentry app pod/deployment (which looks like the default namespace).
So it points that you may have a problem with DNS. You can check by shelling into the sentry app container/pod and trying to ping postgres-sentry:
$ kubectl exec -it <pod-id-of-your-sentry-app> sh
# ping postgres-sentry
Also, check if you have you /etc/resolv.conf looks something like this:
# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
Finally, check if your DNS pods are running:
$ kubectl -n kube-system get pods | grep dns
coredns-xxxxxxxxxxx-xxxxx 1/1 Running 15 116d
coredns-xxxxxxxxxxx-xxxxx 1/1 Running 15 116d
Related
I have gone through the doc mentioned here gitlink as well as doclink
But my job would be a whole lot easier if I could get the dns of a resource type by using any kubernetes command.
Also tried this commands-link
For example, i would like to get the dns name of a service db-service running in dev namespace inside svc.cluster.local
db-service.dev.svc.cluster.local
Any pointers ?
If you need to, you can query that in a pod:
How do I run a container from the command line in Kubernetes (like docker run)?
Retrieve the full name of a service in Kubernetes
using a pod which has some DNS utils
kubectl run tmp-shell --rm -i --tty --image tutum/dnsutils -- /bin/bash
you can then run
root#tmp-shell:/# nslookup db-service
Server: 10.2.32.10
Address: 10.2.32.10#53
Name: db-service.dev.svc.cluster.local
for a one-liner, see: https://serverfault.com/questions/929211/kubernetes-pod-dns-resolution
Try command
kubectl get svc
First column is internal DNS name.
If the type is LoadBalancer then EXTERNAL-IP column will display the external DNS name.
I created a kubernetes cluster on my debian 9 machine using kind.
Which apparently works because I can run kubectl cluster-info with valid output.
Now I wanted to fool around with the tutorial on Learn Kubernetes Basics site.
I have already deployed the app
kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
and started the kubectl proxy.
Output of kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
kubernetes-bootcamp 1/1 1 1 17m
My problem now is: when I try to see the output of the application using curl I get
Error trying to reach service: 'dial tcp 10.244.0.5:80: connect: connection refused'
My commands
export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME/proxy/
For the sake of completeness I can run curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME/ and I get valid output.
The steps from this tutorial module represent environment as if You were working on one of the cluster nodes.
And the command tries to check connectivity to service locally on the node.
However In Your case by running Your kubernetes in a docker (kind) cluster the curl command is most likely ran from the host that is serving the docker containers that have kubernetes in it.
It might be possible to use docker exec to get inside kind node and try to run curl command from there.
Hope this helps.
I'm also doing following the tutorial using kind and got it to work forwarding the port:
kubectl port-forward $POD_NAME 8001:8001
Try add :8080 after the $POD_NAME
curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME:8080/proxy/
I am having an issue with Kubernetes on GKE.
I am unable to resolve services by their name using internal DNS.
this is my configuration
Google GKE v1.15
kubectl get namespaces
NAME STATUS AGE
custom-metrics Active 183d
default Active 245d
dev Active 245d
kube-node-lease Active 65d
kube-public Active 245d
kube-system Active 245d
stackdriver Active 198d
I've deployed a couple of simple services based on openjdk 11 docker image and made with spring boot + actuator in order to have a /actuator/health endpoint to test in dev
kubectl get pods --namespace=dev
NAME READY STATUS RESTARTS AGE
test1-5d86946c49-h9t9l 1/1 Running 0 3h1m
test2-5bb5f4ff8d-7mzc8 1/1 Running 0 3h10m
If i try to execute under
kubectl --namespace=dev exec -it test1-5d86946c49-h9t9 -- /bin/bash
root#test1-5d86946c49-h9t9:/app# cat /etc/resolv.conf
nameserver 10.40.0.10
search dev.svc.cluster.local svc.cluster.local cluster.local europe-west1-b.c.back-office-236415.internal c.back-office-236415.internal google.internal
options ndots:5
root#test1-5d86946c49-h9t9:/app# nslookup test2
Server: 10.40.0.10
Address: 10.40.0.10#53
** server can't find test2: NXDOMAIN
The same issue occurs if I try using test2 service and try to resolve test1. There is a special configuration for namespace to enable DNS resolve? Shouldn't this be automatic?
I have reproduced this using master version 1.15 and and type of service as ‘ClusterIP’. I am able to do look up from the Pod of one service to another. For creating Kubernetes Services in a Google Kubernetes Engine cluster [1] might be helpful.
To see the services:
$ kubectl get svc --namespace=default
To access the deployment:
$ kubectl exec -it [Pod Name] sh
To lookup:
$ nslookup [Service Name]
Every Service defined in the cluster (including the DNS server itself) is assigned a DNS name. By default, a client Pod’s DNS search list will include the Pod’s own namespace and the cluster’s default domain.
“Normal” (not headless) Services are assigned a DNS A record for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.
For “Headless” (without a cluster IP) Services are also assigned a DNS A record for a name.Though this resolves to the set of IPs of the pods selected by the Service.
However, DNS policies can be set on a per-pod basis. Currently Kubernetes supports the following pod-specific DNS policies. These policies are specified in the dnsPolicy field of a Pod Spec [2]:
“Default“: The Pod inherits the name resolution configuration from the node that the pods run on.
“ClusterFirst“: Any DNS query that does not match the configured cluster domain suffix, such as “www.kubernetes.io”, is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured.
“ClusterFirstWithHostNet“: For Pods running with hostNetwork, need to set its DNS policy “ClusterFirstWithHostNet”.
“None“: It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the dnsConfig field in the Pod Spec.
[1]-https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps
[2]-https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config
So this has been working forever. I have a few simple services running in GKE and they refer to each other via the standard service.namespace DNS names.
Today all DNS name resolution stopped working. I haven't changed anything, although this may have been triggered by a master upgrade.
/ambassador # nslookup ambassador-monitor.default
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'ambassador-monitor.default': Try again
/ambassador # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local c.snowcloud-01.internal google.internal
nameserver 10.207.0.10
options ndots:5
Master version 1.14.7-gke.14
I can talk cross-service using their IP addresses, it's just DNS that's not working.
Not really sure what to do about this...
The easiest way to verify if there is a problem with your Kube DNS is to look at the logs StackDriver [https://cloud.google.com/logging/docs/view/overview].
You should be able to find DNS resolution failures in the logs for the pods, with a filter such as the following:
resource.type="container"
("UnknownHost" OR "lookup fail" OR "gaierror")
Be sure to check logs for each container. Because the exact names and numbers of containers can change with the GKE version, you can find them like so:
kubectl get pod -n kube-system -l k8s-app=kube-dns -o \
jsonpath='{range .items[*].spec.containers[*]}{.name}{"\n"}{end}' | sort -u kubectl get pods -n kube-system -l k8s-app=kube-dns
Has the pod been restarted frequently? Look for OOMs in the node console. The nodes for each pod can be found like so:
kubectl get pod -n kube-system -l k8s-app=kube-dns -o \
jsonpath='{range .items[*]}{.spec.nodeName} pod={.metadata.name}{"\n"}{end}'
The kube-dns pod contains four containers:
kube-dns process watches the Kubernetes master for changes in Services and Endpoints, and maintains in-memory lookup structures to serve DNS requests,
dnsmasq adds DNS caching to improve performance,
sidecar provides a single health check endpoint while performing dual health checks (for dnsmasq and kubedns). It also collects dnsmasq metrics and exposes them in the Prometheus format,
prometheus-to-sd scraping the metrics exposed by sidecar and sending them to Stackdriver.
By default, the dnsmasq container accepts 150 concurrent requests. Requests beyond this are simply dropped and result in failed DNS resolution, including resolution for metadata. To check for this, view the logs with the following filter:
resource.type="container"resource.labels.cluster_name="<cluster-name>"resource.labels.namespace_id="kube-system"logName="projects/<project-id>/logs/dnsmasq""Maximum number of concurrent DNS queries reached"
If legacy stackdriver logging of cluster is disabled, use the following filter:
resource.type="k8s_container"resource.labels.cluster_name="<cluster-name>"resource.labels.namespace_name="kube-system"resource.labels.container_name="dnsmasq""Maximum number of concurrent DNS queries reached"
If Stackdriver logging is disabled, execute the following:
kubectl logs --tail=1000 --namespace=kube-system -l k8s-app=kube-dns -c dnsmasq | grep 'Maximum number of concurrent DNS queries reached'
Additionally, you can try to use the command [dig ambassador-monitor.default #10.207.0.10] from each nodes to verify if this is only impacting one node. If it is, you can simple re-create the impacted node.
It appears that I hit a bug that caused the gke-metadata server to start crash pooling (which in turn prevented kube-dns from working).
Creating a new pool with a previous version (1.14.7-gke.10) and migrating to it fixed everything.
I am told a fix has already been submitted.
Thank you for your suggestions.
Start by debugging your kubernetes services [1]. This will tell you whether is a k8s resource issue or kubernetes itself is failing. Once you understand that, you can proceed to fix it. You can post results here if you want to follow up.
[1] https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/
I have kubeDNS set up on a bare metal kubernetes cluster. I thought that would allow me to access services as described here (http:// for those who don't want to follow the link), but when I run
curl https://monitoring-influxdb:8083
I get the error
curl: (6) Could not resolve host: monitoring-influxdb
This is true when I run curl on a service name in any namespace. Is this an error with my kubDNS setup or are there different steps I need to take in order to achieve this? I get the expected output when I run the test at the end of this article.
For reference:
kubeDNS controller yaml files
kubeDNS service yaml file
kubelet flags
output of kubectl get svc in default and kube-system namespaces
The service discovery that you're trying to is documented at https://kubernetes.io/docs/concepts/services-networking/dns-pod-service, and is for communications within one pod talking to an existing service, not from nodes (or the master) to speak to Kubernetes services.
You will want to leverage the DNS for the service in form of <servicename>.<namespace> or <servicename>.<namespace>.svc.cluster.local. To see this in operation, kick up an interactive pod with busybox (or use an existing pod of your own) with something like:
kubectl run -i --tty alpine-interactive --image=alpine --restart=Never
and within that shell that is provided there, make an nslookup command. From your example, I'm guessing you're trying to access influxDB from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb, then it will be installed into the kube-system namespace, and the service name you'd use from another Pod internally to the cluster would be:
monitoring-influxdb.kube-system.svc.cluster.local
For example:
kubectl run -i --tty alpine --image=alpine --restart=Never
If you don't see a command prompt, try pressing enter.
/ # nslookup monitoring-influxdb.kube-system.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: monitoring-influxdb.kube-system.svc.cluster.local
Address 1: 10.102.27.233 monitoring-influxdb.kube-system.svc.cluster.local
As #Michael Hausenblas pointed out in the comments, curl http://monitoring-influxdb:8086 needs to be run from within a pod. Doing that provided the expected results