Ingress responding with 'default backend - 404' when using GKE - kubernetes

Question

Using the latest Kubernetes version in GCP (1.6.4), I have the following Ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myproject
namespace: default
annotations:
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: "gce"
spec:
rules:
- host: staging.myproject.io
http:
paths:
- path: /poller
backend:
serviceName: poller
servicePort: 8080
Here is my service and deployment:
apiVersion: v1
kind: Service
metadata:
name: poller
labels:
app: poller
tier: backend
role: service
spec:
type: NodePort
selector:
app: poller
tier: backend
role: service
ports:
- port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: poller
spec:
replicas: 1
template:
metadata:
labels:
app: poller
tier: backend
role: service
spec:
containers:
- name: poller
image: gcr.io/myproject-1364/poller:latest
imagePullPolicy: Always
env:
- name: SPRING_PROFILES_ACTIVE
value: staging
- name: GET_HOSTS_FROM
value: dns
ports:
- containerPort: 8080
In my /etc/hosts I have a line like:
35.190.37.148 staging.myproject.io
However, I get default backend - 404 when curling any endpoint on staging.myproject.io:
$ curl staging.myproject.io/poller/cache/status
default backend - 404
I have the exact same configuration working locally inside Minikube, with the only difference being the domain (dev.myproject.io), and that works like a charm.
I have read and tried pretty much everything that I could find, including stuff from here and here and here, but maybe I'm just missing something... any ideas?

Answer 1

It does take 5-10 minutes for an Ingress to actually become usable in GKE. In the meanwhile, you can see responses with status codes 404, 502 and 500.
There is an ingress tutorial here: https://cloud.google.com/container-engine/docs/tutorials/http-balancer I recommend following it. Based on what you pasted, I can say the following:
You use service.Type=NodePort, which is correct.
I am not sure about the ingress.kubernetes.io/rewrite-target annotation, maybe that's the issue.
Make sure your application responds 200 OK to GET / request.
Also I realize you curl http://<ip>/ but your Ingress spec only handles /poller endpoint. So it's normal you get default backend - 404 response while querying /. You didn't configure any backends for / path in your Ingress spec.

Answer 2

If anyone else is facing this problem, check if header Host is correct and matches expected domain.