Website responding with "default backend - 404" on GKE - Kubernetes - kubernetes

Question

On one of our domains https://www.secretwish.in - all the 404 traffic is going to the default GKE Ingress whereas it has to route to my application.
All other pages on my application are working fine, the problem is just with 404 page, all traffic is going on gke default ingress. Sample URL - https://www.secretwish.in/hshshs
Need to find a solution for this so that all traffic starts routing to my app
Cluster Version - 1.14.10-gke.27
The ingress file looks like:-
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ans-commerce-ingress
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/tls-minimum-version: "1.0"
spec:
tls:
- hosts:
- www.secretwish.in
secretName: www-secretwish-in-tls
- host: www.secretwish.in
http:
paths:
- path: /
backend:
serviceName: ans-commerce
servicePort: 80

Answer 1

In GKE docs you can find information regarding GKE Ingress, that for specific path you should specify backend, otherwise you will received issue 404 default backend.
You can specify a default backend by providing a backend field in your
Ingress manifest. Any requests that don't match the paths in the rules
field are sent to the Service and port specified in the backend field.
... If you don't specify a default backend, GKE provides a default
backend that returns 404.
Default backend will redirect all request which could not be found
in any spec.rules.http.paths.path
For little test I've used 2 deployments and 2 services form this gke example.
Option 1 without configured default end
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- http:
paths:
- path: /world
backend:
serviceName: hello-world
servicePort: 60000
- path: /kube
backend:
serviceName: hello-kubernetes
servicePort: 80
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176
default backend - 404
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/kube
Hello Kubernetes!
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/world
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/yvgbhuij
default backend - 404
Option 2 With defailt backend
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: hello-world
servicePort: 60000
rules:
- http:
paths:
- path: /world
backend:
serviceName: hello-world
servicePort: 60000
- path: /kube
backend:
serviceName: hello-kubernetes
servicePort: 80
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/hello
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-kd6fg
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/kube
Hello Kubernetes!
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/fyvgbhujnkl
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
Please keep in mind that Ingress on GKE needs about 5-6 minutes before it starts working properly

Answer 2

Not sure our solution applies in you case. We also faced similar issue(not exact), we didn't want to change the deployed ingress(i.e. can't add server-alias) So we some how made sure that all host address typed in browser/request translates to what is configured in ingress.
In this example, we would have created entry in local /etc/hosts
<actual ingress IP> www.secretwish.in