Ingress on GKE. Run specify service healthcheck - kubernetes

Question

I try to use ingress for loadbalancer of 2 services on Google Kubernetes engine:
here is ingress config for this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: basic-ingress
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: web
servicePort: 8080
- path: /v2/keys
backend:
serviceName: etcd-np
servicePort: 2379
where web is some example service from google samples:
apiVersion: v1
kind: Service
metadata:
name: web
namespace: default
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
run: web
type: NodePort
----
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: default
spec:
selector:
matchLabels:
run: web
template:
metadata:
labels:
run: web
spec:
containers:
- image: gcr.io/google-samples/hello-app:1.0
imagePullPolicy: IfNotPresent
name: web
ports:
- containerPort: 8080
protocol: TCP
But the second service is ETCD cluster with NodePort service:
---
apiVersion: v1
kind: Service
metadata:
name: etcd-np
spec:
ports:
- port: 2379
targetPort: 2379
selector:
app: etcd
type: NodePort
But only first ingress rule works properly i see in logs:
ingress.kubernetes.io/backends: {"k8s-be-30195--ebfd7339a961462d":"UNHEALTHY","k8s-be-30553--ebfd7339a961462d":"HEALTHY","k8s-be-31529--ebfd7339a961462d":"HEALTHY"}
I etcd-np works properly it is not a problem of etcd , i think that the problem is that etcd server answers with 404 on GET / request and some healthcheck on ingress level does not allow to use it .
Thats why i have 2 questions :
1 ) How can I provide healthcheck urls for each backend path on ingress
2 ) How can I debug such issues . What I see now is
kubectl describe ingress basic-ingress
Name: basic-ingress
Namespace: default
Address: 4.4.4.4
Default backend: default-http-backend:80 (10.52.6.2:8080)
Rules:
Host Path Backends
---- ---- --------
*
/* web:8080 (10.52.8.10:8080)
/v2/keys etcd-np:2379 (10.52.0.2:2379,10.52.2.4:2379,10.52.8.4:2379)
Annotations: ingress.kubernetes.io/backends:
{"k8s-be-30195--ebfd7339a961462d":"UNHEALTHY","k8s-be-30553--ebfd7339a961462d":"HEALTHY","k8s-be-31529--ebfd7339a961462d":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-basic-ingress--ebfd7339a961462d
ingress.kubernetes.io/target-proxy: k8s-tp-default-basic-ingress--ebfd7339a961462d
ingress.kubernetes.io/url-map: k8s-um-default-basic-ingress--ebfd7339a961462d
Events: <none>
But it does not provide me any info about this incident
UP
kubectl describe svc etcd-np
Name: etcd-np
Namespace: default
Labels: <none>
Annotations: Selector: app=etcd
Type: NodePort
IP: 10.4.7.20
Port: <unset> 2379/TCP
TargetPort: 2379/TCP
NodePort: <unset> 30195/TCP
Endpoints: 10.52.0.2:2379,10.52.2.4:2379,10.52.8.4:2379
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>

Answer 1

According to the doc.
A Service exposed through an Ingress must respond to health checks
from the load balancer. Any container that is the final destination of
load-balanced traffic must do one of the following to indicate that it
is healthy:
Serve a response with an HTTP 200 status to GET requests on the / path.
Configure an HTTP readiness probe. Serve a response with an HTTP 200 status to GET requests on the path specified by the readiness
probe. The Service exposed through an Ingress must point to the same
container port on which the readiness probe is enabled.
For example, suppose a container specifies this readiness probe:
...
readinessProbe:
httpGet:
path: /healthy
Then if the handler for the container's /healthy path returns an
HTTP 200 status, the load balancer considers the container to be
alive and healthy.
Now since ETCD has a health endpoint at /health the readiness probe will look like
...
readinessProbe:
httpGet:
path: /health
This becomes a bit tricky if mTLS is enabled in ETCD. To avoid that check the docs.