Istio on minikube - envoy missing listener for inbound application port: 9095 - kubernetes

I follow this istio tutorial (part 3). After I created minikube local registry, I need to run the following command:
kubectl run hellodemo --image=hellodemo:v1 --port=9095 --image-pull-policy=IfNotPresent
Which should run image and istio-proxy on the Pod.
When I run kubectl get pods, I get:
NAME READY STATUS RESTARTS AGE
hellodemo-6d49fc6c51-adsa1 1/2 Running 0 1h
When I run kubectl logs hellodemo-6d49fc6c51-adsa1 istio-proxy:
* failed checking application ports. listeners="0.0.0.0:15090","10.110.201.202:16686","10.96.0.1:443","10.104.103.28:15443","10.104.103.28:15031","10.101.128.212:14268","10.104.103.28:15030","10.111.177.172:443","10.104.103.28:443","10.109.4.23:80","10.111.177.172:15443","10.104.103.28:15020","10.104.103.28:15032","10.105.175.151:15011","10.101.128.212:14267","10.96.0.10:53","10.104.103.28:31400","10.104.103.28:15029","10.98.84.0:443","10.99.194.141:443","10.99.175.237:42422","0.0.0.0:9411","0.0.0.0:3000","0.0.0.0:15010","0.0.0.0:15004","0.0.0.0:8060","0.0.0.0:9901","0.0.0.0:20001","0.0.0.0:8080","0.0.0.0:9091","0.0.0.0:80","0.0.0.0:15014","0.0.0.0:9090","172.17.0.6:15020","0.0.0.0:15001"
* envoy missing listener for inbound application port: 9095
2019-05-02T16:24:28.709972Z info Envoy proxy is NOT ready: 2 errors occurred:
* failed checking application ports. listeners="0.0.0.0:15090","10.110.201.202:16686","10.96.0.1:443","10.104.103.28:15443","10.104.103.28:15031","10.101.128.212:14268","10.104.103.28:15030","10.111.177.172:443","10.104.103.28:443","10.109.4.23:80","10.111.177.172:15443","10.104.103.28:15020","10.104.103.28:15032","10.105.175.151:15011","10.101.128.212:14267","10.96.0.10:53","10.104.103.28:31400","10.104.103.28:15029","10.98.84.0:443","10.99.194.141:443","10.99.175.237:42422","0.0.0.0:9411","0.0.0.0:3000","0.0.0.0:15010","0.0.0.0:15004","0.0.0.0:8060","0.0.0.0:9901","0.0.0.0:20001","0.0.0.0:8080","0.0.0.0:9091","0.0.0.0:80","0.0.0.0:15014","0.0.0.0:9090","172.17.0.6:15020","0.0.0.0:15001"
* envoy missing listener for inbound application port: 9095
2019-05-02T16:24:30.729987Z info Envoy proxy is NOT ready: 2 errors occurred:
* failed checking application ports. listeners="0.0.0.0:15090","10.110.201.202:16686","10.96.0.1:443","10.104.103.28:15443","10.104.103.28:15031","10.101.128.212:14268","10.104.103.28:15030","10.111.177.172:443","10.104.103.28:443","10.109.4.23:80","10.111.177.172:15443","10.104.103.28:15020","10.104.103.28:15032","10.105.175.151:15011","10.101.128.212:14267","10.96.0.10:53","10.104.103.28:31400","10.104.103.28:15029","10.98.84.0:443","10.99.194.141:443","10.99.175.237:42422","0.0.0.0:9411","0.0.0.0:3000","0.0.0.0:15010","0.0.0.0:15004","0.0.0.0:8060","0.0.0.0:9901","0.0.0.0:20001","0.0.0.0:8080","0.0.0.0:9091","0.0.0.0:80","0.0.0.0:15014","0.0.0.0:9090","172.17.0.6:15020","0.0.0.0:15001"
* envoy missing listener for inbound application port: 9095
Do you know what is the problem that prevent the istio-proxy container to come up?
I use istio-1.1.4 on minikube.


I was also having the same problem. I followed the documentation, and it said to enable SDS in the gateway. However, I enabled it in the gateway and also at the global scope as well, and this caused the error above.
I removed the following code from my values.yml file and everything worked:
global:
sds:
enabled: true

Related

Liveness-Probe of one pod via another

On my Kubernetes Setup, I have 2 pods - A (via deployment) and B(via DS).
Pod B is somehow dependent on Pod A being fully started through. I would now like to set an HTTP Liveness-Probe in Pods B, to restart POD B if health check via POD A fails. Restarting works fine if I put the External IP of my POD A's service in the host. The issue is in resolving DNS name in the host.
It works if I set it like this:
livenessProbe:
httpGet:
host: <POD_A_SERVICE_EXTERNAL_IP_HERE>
path: /health
port: 8000
Fails if I set it like this:
livenessProbe:
httpGet:
host: auth
path: /health
port: 8000
Failed with following error message:
Liveness probe failed: Get http://auth:8000/health: dial tcp: lookup auth on 8.8.8.8:53: no such host
ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Is the following line on the above page true for HTTP Probes as well?
"you can not use a service name in the host parameter since the kubelet is unable to resolve it."

Correct 👍, DNS doesn't work for liveness probes, the kubelet network space cannot basically resolve any in-cluster DNS.
You can consider putting both of your services in a single pod as sidecars. This way they would share the same address space if one container fails then the whole pod is restarted.
Another option is to create an operator 🔧 for your pods/application and basically have it check the liveness through the in-cluster DNS for both pods separately and restart the pods through the Kubernetes API.
You can also just create your own script in a pod that just calls curl to check for a 200 OK and kubectl to restart your pod if you get something else.
Note that for the 2 options above you need to make sure that Coredns is stable and solid otherwise your health checks might fail to make your services have potential downtime.
✌️☮️

CRD probe failing

I am installing service catalog which uses CRD and have created the same. Now I am running my controller deployment file and the image running in it runs a CRD list command to verify CRD are in place. This use to work fine previously but now CRD Probe is failing with error:
1226 07:45:01.539118 1 round_trippers.go:438] GET https://169.72.128.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?labelSelector=svcat%3Dtrue in 30000 milliseconds
I1226 07:45:01.539158 1 round_trippers.go:444] Response Headers:
Error: while waiting for ready Service Catalog CRDs: failed to list CustomResourceDefinition: Get https://169.72.128.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?labelSelector=svcat%3Dtrue: dial tcp 169.72.128.1:443: i/o timeout
I have followed same steps as previously but could not debug now.
Inside the controller code it is trying to make following call:
list, err := r.client.ApiextensionsV1beta1().CustomResourceDefinitions().List(v1.ListOptions{LabelSelector: labels.SelectorFromSet(labels.Set{"svcat": "true"}).String()})
Which is failing.
Update 1 : Installation works fine in default namespace but fails in specific namespace.
Environment Info: On Prem k8s cluster, latest k8s, 2 node cluster.

It's not a port issue.Service accounts use 443 port to connect to Kubernetes API Server. Check the if there is any network policy blocking the communication between your namespace and Kube-System namespace.

TLS errors in pod logs in Kubernetes

I use Kubernetes version 1.15.3 .
I use calico as cni plugin.
I have a deployment that has service configured to as Nodeport on a port serviced by HTTPS protocol.
For some reason when I look at one of the pods of the deployment I see this output:
2020-01-07T08:45:02.905Z [INFO] http: TLS handshake error from 10.233.92.0:60286: tls:first record does not look like a TLS handshake
2020-01-07T08:44:42.884Z [INFO] http: TLS handshake error from 10.82.83.213:45654: tls: first record does not look like a TLS handshake
The IP 10.82.83.213 is the IP of the node and the second IP is some internal IP of Kubernetes.
These errors are not interfering with the actual work of hte pod since it is up but I makes it hard to look for pother errors since this lines appears every 4 seconds.
Initially I tried removing the Probes but that did not help.
Once I remove the NodePort service or changes the label these error stops but ofcourse then the pod is not reachable.
Does any one knows the root cause for this?
I would assume kube-proxy but I couldn't find evidence of that.
thanks

How do we debug networking issues within istio pods?

I am working on setting up istio in my kubernetes cluster.
I downloaded istio-1.4.2 and installed demo profile and did manual sidecar injection.
But when I check sidecar pod logs, I am getting the below error.
2019-12-26T08:54:17.694727Z error k8s.io/client-go#v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1beta1.MutatingWebhookConfiguration: Get https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
It seems to be the networking issue, but could you please let me know what it is trying to do exactly?
Is there a way to get more logs than just 'connection refused'?
How do we verify networking issues between istio pods. It seems I cannot run 'wget', 'curl', 'tcpdump', 'netstat' etc within istio sidecar pod to debug further.
All the pods in kube-system namespace are working fine.

Check what port your API Server is serving https traffic(controlled by this flag --secure-port int Default: 6443). It may be 6443 instead of 443.
Check what is the value of server in your kubeconfig and are you able to connect to your kubernetes via kubectl using that kubeconfig.
Another thing to check is whether you have network policy attached to the namespace which blocks egress traffic.
And you could use an ephemeral container to debug issue with the sidecar
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
https://github.com/aylei/kubectl-debug

Readiness-Probe another Service on boot-up of Pod

On my Kubernetes Setup, i have 2 Services - A and B.
Service B is dependent on Service A being fully started through.
I would now like to set a TCP Readiness-Probe in Pods of Service B, so they test if any Pod of Service A is fully operating.
the ReadinessProbe section of the deployment in Service B looks like:
readinessProbe:
tcpSocket:
host: serviceA.mynamespace.svc.cluster.local
port: 1101 # same port of Service A Readiness Check
I can apply these changes, but the Readiness Probe fails with:
Readiness probe failed: dial tcp: lookup serviceB.mynamespace.svc.cluster.local: no such host
I use the same hostname on other places (e.g. i pass it as ENV to the container) and it works and gets resolved.
Does anyone have an idea to get the readiness working for another service or to do some other kind of dependency-checking between services?
Thanks :)

Due to the fact that Readiness and Liveness probes are fully managed by kubelet node agent and kubelet inherits DNS discovery service from the particular Node configuration, you are not able to resolve K8s internal nameserver DNS records:
For a probe, the kubelet makes the probe connection at the node, not
in the pod, which means that you can not use a service name in the
host parameter since the kubelet is unable to resolve it.
You can consider scenario when your source Pod A consumes Node IP Address by propagating hostNetwork: true parameter, thus kubelet can reach and success Readiness probe from within Pod B, as described in the official k8s documentation:
tcpSocket:
host: Node Hostname or IP address where Pod A residing
port: 1101
However, I've found Stack thread, where you can get more efficient solution how to achieve the same result through Init Containers.

In addition to Nick_Kh's answer, another workaround is to use probe by command, which is executed in a container.
To perform a probe, the kubelet executes the command cat /tmp/healthy in the target container. If the command succeeds, it returns 0, and the kubelet considers the container to be alive and healthy.
An example:
readinessProbe:
exec:
command:
- sh
- -c
- wget -T2 -O- http://service