Traefik shows 404 to everywhere except dashboard - Metallb - kubernetes

Question

I try to setup the ingress with traefik but no luck. I would not use TLS just simply the port 80. I have a service with port 8080. If I curl to that service from inside of the cluster it works well. I get the HTTP/200. But if I would connect to he path externally it doesnt working.
The Traefik dashboard works well on port 8080
Im using the following setup
Traefik: 1.7.7
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik-ingress-controller
namespace: default
labels:
k8s-app: traefik-ingress-lb
spec:
replicas: 1
selector:
matchLabels:
k8s-app: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- image: traefik:v1.7.7
name: traefik-ingress-lb
ports:
- name: http
containerPort: 80
- name: admin
containerPort: 8080
args:
- --api
- --web
- --kubernetes
- --logLevel=DEBUG
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: default
annotations:
metallb.universe.tf/address-pool: mmas-ip-space
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 8080
name: admin
type: LoadBalancer
I have a service what running on port 8080 and I created an ingress rule for it. This is a test service
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: webmust-ing
namespace: default
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- http:
paths:
- path: /helloservice
backend:
serviceName: hellok8s-service
servicePort: 8080
I get the 404 with curl to /helloservice or try to open from a browser
curl -v http://10.24.33.32/helloservice
curl -v http://10.24.33.32:8080/helloservice
I if curl to the service's ip address directly inside of the cluster, I get the 200/OK
curl -v http://10.100.168.2:8080
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hellok8s-service ClusterIP 10.100.168.2 <none> 8080/TCP 5d7h
cat helloservice.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hellok8s-deployment
labels:
app: hellok8s
spec:
selector:
matchLabels:
app: hellok8s
template:
metadata:
labels:
app: hellok8s
spec:
containers:
- name: hellok8s
image: docker.io/rlkamradt/hellok8s:latest
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: hellok8s-service
spec:
type: ClusterIP
selector:
app: hellok8s
ports:
- port: 8080
targetPort: 8080

Answer 1

Finally I fixed the problem with the following change in the ingress.
annotations:
ingress.kubernetes.io/protocol: http
traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip