GKE Ingress with Multiple Backend Services returns 404 - kubernetes

Question

I'm trying to create a GKE Ingress that points to two different backend services based on path. I've seen a few posts explaining this is only possible with an nginx Ingress because gke ingress doesn't support rewrite-target. However, this Google documentation, GKE Ingresss - Multiple backend services, seems to imply otherwise. I've followed the steps in the docs but haven't had any success. Only the service that is available on the path prefix of / is returned. Any other path prefix, like /v2, returns a 404 Not found.
Details of my setup are below. Is there an obvious error here -- is the Google documentation incorrect and this is only possible using nginx ingress?
-- Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: app-static-ip
networking.gke.io/managed-certificates: app-managed-cert
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /v2
pathType: Prefix
backend:
service:
name: api-2-service
port:
number: 8080
-- Service 1
apiVersion: v1
kind: Service
metadata:
name: api-service
labels:
app: api
spec:
type: NodePort
selector:
app: api
ports:
- port: 80
targetPort: 5000
-- Service 2
apiVersion: v1
kind: Service
metadata:
name: api-2-service
labels:
app: api-2
spec:
type: NodePort
selector:
app: api-2
ports:
- port: 8080
targetPort: 5000

Answer 1

GCP Ingress supports multiple paths. This is also well described in Setting up HTTP(S) Load Balancing with Ingress. For my test I've used both Hello-world v1 and v2.
There are 3 possible issues.
Issue is with container ports opened. You can check it using netstat:
$ kk exec -ti first-55bb869fb8-76nvq -c container -- bin/sh
/ # netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::8080 :::* LISTEN 1/hello-app
Issue might be also caused by the Firewall configuration. Make sure you have proper settings. (In general, in the new cluster I didn't need to add anything but if you have more stuff and have specific Firewall configurations it might block).
Misconfiguration between port, containerPort and targetPort.
Below my example:
1st deployment with
apiVersion: apps/v1
kind: Deployment
metadata:
name: first
labels:
app: api
spec:
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: container
image: gcr.io/google-samples/hello-app:1.0
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api-service
labels:
app: api
spec:
type: NodePort
selector:
app: api
ports:
- port: 5000
targetPort: 8080
2nd deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: second
labels:
app: api-2
spec:
selector:
matchLabels:
app: api-2
template:
metadata:
labels:
app: api-2
spec:
containers:
- name: container
image: gcr.io/google-samples/hello-app:2.0
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api-2-service
labels:
app: api-2
spec:
type: NodePort
selector:
app: api-2
ports:
- port: 6000
targetPort: 8080
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 5000
- path: /v2
pathType: Prefix
backend:
service:
name: api-2-service
port:
number: 6000
Outputs:
$ curl 35.190.XX.249
Hello, world!
Version: 1.0.0
Hostname: first-55bb869fb8-76nvq
$ curl 35.190.XX.249/v2
Hello, world!
Version: 2.0.0
Hostname: second-d7d87c6d8-zv9jr
Please keep in mind that you can also use Nginx Ingress on GKE by adding specific annotation.
kubernetes.io/ingress.class: "nginx"
Main reason why people use nginx ingress on GKE is using rewrite annotation and possibility to use ClusterIP or NodePort as serviceType, where GCP ingress allows only NodePort serviceType.
Additional information you can find in GKE Ingress for HTTP(S) Load Balancing