One Deployment is reported with unhealthy upstream - kubernetes

Question

Using Istio with Kubernetes and facing an issue with a simple routerule splitting traffic between two Deployments. Only one of the Deployment (howdy) is returning results correctly. The other Deployment (hello) is reporting "no healthy upstream".
Here is the k8s manifest:
apiVersion: v1
kind: Service
metadata:
name: greeting
labels:
name: greeting
spec:
selector:
app: greeting
ports:
- name: http
protocol: TCP
port: 8081
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: greeting-hello
labels:
name: greeting-hello
spec:
replicas: 1
template:
metadata:
labels:
app: greeting
greeting: hello
spec:
containers:
- name: greeting
image: arungupta/greeting:hello
imagePullPolicy: Always
ports:
- containerPort: 8081
readinessProbe:
httpGet:
path: /resources/greeting
port: 8081
initialDelaySeconds: 50
periodSeconds: 5
--- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: greeting-howdy
labels:
name: greeting-howdy
spec:
replicas: 1
template:
metadata:
labels:
app: greeting
greeting: howdy
spec:
containers:
- name: greeting
image: arungupta/greeting:howdy
imagePullPolicy: Always
ports:
- containerPort: 8081
readinessProbe:
httpGet:
path: /resources/greeting
port: 8081
initialDelaySeconds: 50
periodSeconds: 5
And the route is:
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: greeting-50-50
spec:
destination:
name: greeting
route:
- labels:
app: greeting
greeting: hello
weight: 100
- labels:
app: greeting
greeting: howdy
weight: 0
Any idea?
This is also tracked at https://github.com/aws-samples/aws-microservices-deploy-options/issues/239

Answer 1

I've reproduced your deployment issue and found the following errors during the process:
Warning Unhealthy 48m (x7 over 49m) kubelet, gke-cluster-1-default-pool-e44042f6-kndh Readiness probe failed: Get http://10.0.2.30:8081/resources/greeting: dial tcp 10.0.2.30:8081: getsockopt: connection refused
I would recommend you to check if docker containers in this particular configuration are
exposing endpoints on the specified (8081) port. I also found that pods are not registered in any
of the endpoints:
$ kubectl describe service
[...]
IP: 10.3.247.97
Port: http 8081/TCP
TargetPort: 8081/TCP
Endpoints:
Session Affinity: None
Events: <none>
The problem is that the application did not bind to the port which was set as a container port,
and the application did not receive connection.