How to make harbor reachable behind istio ingress? - kubernetes

I have installed Harbor as follows:
helm install hub harbor/harbor \
--version 1.3.2 \
--namespace tool \
--set expose.ingress.hosts.core=hub.service.example.io \
--set expose.ingress.annotations.'kubernetes\.io/ingress\.class'=istio \
--set expose.ingress.annotations.'cert-manager\.io/cluster-issuer'=letsencrypt-prod \
--set externalURL=https://hub.service.example.io \
--set notary.enabled=false \
--set secretkey=secret \
--set harborAdminPassword=pw
Everything is up and running but the page is not reachable via https://hub.service.example.io. The same problem occurs here Why css and png are not accessible? but how to set wildcard * in Helm?
Update
Istio supports ingress gateway. This for example works without Gateway and VirtualService definition:
apiVersion: v1
kind: Service
metadata:
name: hello-kubernetes-first
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
selector:
app: hello-kubernetes-first
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-kubernetes-first
spec:
replicas: 3
selector:
matchLabels:
app: hello-kubernetes-first
template:
metadata:
labels:
app: hello-kubernetes-first
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.8
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: Hello from the first deployment!
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: istio
name: helloworld-ingress
spec:
rules:
- host: "hw.service.example.io"
http:
paths:
- path: "/*"
backend:
serviceName: hello-kubernetes-first
servicePort: 80
---

I would say it won't work with ingress and istio.
As mentioned here
Simple ingress specifications, with host, TLS, and exact path based matches will work out of the box without the need for route rules. However, note that the path used in the ingress resource should not have any . characters.
For example, the following ingress resource matches requests for the example.com host, with /helloworld as the URL.
$ kubectl create -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: simple-ingress
annotations:
kubernetes.io/ingress.class: istio
spec:
rules:
- host: example.com
http:
paths:
- path: /helloworld
backend:
serviceName: myservice
servicePort: grpc
EOF
However, the following rules will not work because they use regular expressions in the path and ingress.kubernetes.io annotations:
$ kubectl create -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: this-will-not-work
annotations:
kubernetes.io/ingress.class: istio
# Ingress annotations other than ingress class will not be honored
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: example.com
http:
paths:
- path: /hello(.*?)world/
backend:
serviceName: myservice
servicePort: grpc
EOF
I assume your hello-world is working because of just 1 annotation which is ingress class.
If you take a look at annotations of harbor here, it might be the problem when you want to use ingress with istio.
but how to set wildcard * in Helm?
Wildcard have nothing to do here. As I mentioned in this answer you can use either wildcard or additional paths, which is done well. Take a look at the ingress paths here.

https://github.com/goharbor/harbor-helm/blob/master/templates/ingress/ingress.yaml#L5
If you look here, they have the path hardcoded to a couple ingress options. Envoy/istio isn't one of them. However, you may be in luck- expose.ingress.controller set to "gce" seems to set the paths the way you need them to be. (I've never used gce, maybe they even use istio?)
Edit- original answer is below. Apparently there is an ingress controller you can enable in istio. There are absolutely no docs on it, but what should I expect?
In your case though, helm is not your problem. istio doesn't use ingress objects, it uses 'Gateways' and 'VirtualServices'. You can't configure an app to use the istio ingress system using kubernetes.io/ingress.class annotations.
(at least, that has been my experience, and I can't find anything to contradict that in their docs, but it is completely possible there is an istio ingress controller tha

Related

Deploying Jaeger in a url different than root

I am trying to deploy Jaeger all-in-one image in a kubernetes cluster.
Jaeger is not in the root of the url, meaning it's accessible through https://somedomain.com/xyz/jaeger
I have an ingress rule which seems to be pointing correctly to a Service which is also referencing fine the pod in a deployment (I can see all this in Rancher UI).
But somehow when I try to access, nginx is throwing a 502 Bad Gateway error.
This is how the ingress rule looks like
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
namespace: my-namespace
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: somedomain.com
http:
# Jaeger
- path: /xyz/jaeger(/|$)(.*)
pathType: Prefix
backend:
service:
name: jaeger
port:
number: 16868
Then in the pod definition I tried using the QUERY_BASE_PATH env var setting it to /xyz/jaeger but that made no difference at all.
The problem was an incorrect port being specified.
16868 instead of 16686

can't reach to my grafana dashboard on k8s cluster using ingress from browser

I've installed Prometheus and Grafana on my Kubernetes cluster using helm:
$helm install prometheus prometheus-community/kube-prometheus-stack
All the pods, deployments and services are up and running. When I use port-forwarding like this:
kubectl port-forward deployment/prometheus-grafana 3000
I can reach my grafana dashboard using browser but when I want to use ingress instead of port-forward the response is:
and I can't reach to Grafana dashboard.
My ingress yaml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana-ingress
namespace: default
spec:
ingressClassName: kong
rules:
- http:
paths:
- path: /grafana/login
pathType: Prefix
backend:
service:
name: prometheus-grafana
port:
number: 80
and the prometheus-grafana service yaml file is :
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: default
creationTimestamp: "2021-09-15T11:07:30Z"
labels:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: grafana
app.kubernetes.io/version: 8.1.2
helm.sh/chart: grafana-6.16.4
name: prometheus-grafana
namespace: default
resourceVersion: "801373"
uid: e1f57de9-94d0-460a-a427-4a97fd770e12
spec:
clusterIP: 10.100.90.147
clusterIPs:
- 10.100.90.147
ports:
- name: service
port: 80
protocol: TCP
targetPort: 3000
selector:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: grafana
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
I have posted community wiki answer for better visibility. The problem is solved and it was related to ingress yaml file.
Solution:
I changed my ingress file : added host: grafana.example.com and changed path to / . Everything works smoothly.
The suggestion of the solution:
What I mean is, the Ingress defines only one path, /grafana/login with type Prefix. Surely Grafana will have other paths other than /grafana/login so first thing I'd try is to just use a single path, /grafana. When you use port forward, at which path can you open the grafana dashboard? Because Grafana will probably expects requests to arrive at that same path / paths.
Explanation:
Grafana is a web app and it is expecting to be served directly under the root path of the server. You need to expose it under / as a path, use rewrite target rules or serve it under a subdomain of your host. Try if, with path /, everything works as expected or not as a first thing

Generating a redirect with traefik ingress on k3s?

I'm running prometheus and grafana under k3s, accessible (respectively) at http://monitoring.internal/prometheus and http://monitoring.internal/grafana. The grafana Ingress object, for example, looks like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: monitoring.internal
http:
paths:
- path: /grafana
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
This works fine, except that if you land at
http://monitoring.internal/, you get a 404 error. I would like
requests for http://monitoring.internal/ to redirect to
http://monitoring.internal/grafana. I could perhaps create another
service that runs something like darkhttpd ... --forward-all http://monitoring.internal/grafana, and create an Ingress object
that would map / to that service, but it seems like there ought to
be a way to do this with Traefik itself.
It looks like I'm running Traefik 2.4.8 locally:
$ kubectl -n kube-system exec -it deployment/traefik -- traefik version
Version: 2.4.8
Codename: livarot
Go version: go1.16.2
Built: 2021-03-23T15:48:39Z
OS/Arch: linux/amd64
I've found this documentation for 1.7 that suggests there is an annotation for exactly this purpose:
traefik.ingress.kubernetes.io/app-root: "/index.html": Redirects
all requests for / to the defined path.
But setting that on the grafana ingress object doesn't appear to have
any impact, and I haven't been able to find similar docs for 2.x
(I've looked around
here, for
example).
What's the right way to set up this sort of redirect?
Since I haven't been able to figure out traefik yet, I thought I'd post my solution here in case anyone else runs into the same situation. I am hoping someone comes along who knows The Right Way to to do this, and if I figure out I'll update this answer.
I added a new deployment that runs darkhttpd as a simple director:
apiVersion: apps/v1
kind: Deployment
metadata:
name: redirector
spec:
replicas: 1
template:
spec:
containers:
- name: redirector
image: docker.io/alpinelinux/darkhttpd
ports:
- containerPort: 8080
args:
- --forward-all
- http://monitoring.internal/grafana
A corresponding Service:
apiVersion: v1
kind: Service
metadata:
name: redirector
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
And the following Ingress object:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: redirector
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: monitoring.internal
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: redirector
port:
number: 8080
These are all deployed with kustomize, which takes care of
adding labels and selectors in the appropriate places. The
kustomization.yaml look like:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- ingress.yaml
- service.yaml
commonLabels:
component: redirector
With all this in place, requests to http://monitoring.internal/ hit the redirector pod.

How to make Traefik compatible with Microk8s

I have a working setup on Minikube with Traefik as ingress controller. I tried to use that setup on Microk8s but Traefik is not able to work and although I can see the Traefik dashboard and it says that everything is working but every time I try to use the ingress urls I face timeout but if I use the endpoint IP of that service (which I can see in the traefik dashboard) I am able to access to that service but not fully. I can have access to IP/service1 but I can't have access to any of its sub urls, IP/service1/sub-service1 not working.
I also tried microk8s.enable ingress but it created an nginx ingress for me and then I disabled it because I want to use traefik.
Do I need to change my configuration so it becomes compatible with Microk8s? If yes how?
I have to mention that I have two ingress files:
traefik-ui.yaml: which contains both the service and ingress for my traefik. I use this service+ingress to access the traefik dashboard and as I mentioned it works
wws-ingress.yaml: is my main ingress which enables the communication with my components inside kubernetes and this is the part that doesn't work.
My yaml files:
traefik-ui.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- name: web
port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
rules:
- host: traefik-ui.minikube
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
wws-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wws
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.frontend.rule.type: PathPrefixStrip
traefik.frontend.passHostHeader: "true"
traefik.backend.loadbalancer.sticky: "true"
#traefik.ingress.kubernetes.io/rule-type: ReplacePathRegex
traefik.wss.protocol: http
traefik.wss.protocol: https
spec:
rules:
- host: streambridge.local
http:
paths:
- path: /streambridge
backend:
serviceName: streambridge
servicePort: 9999
- path: /dashboard
backend:
serviceName: dashboard
servicePort: 9009
- path: /gateway
backend:
serviceName: gateway
servicePort: 8080
- path: /rdb
backend:
serviceName: rethinkdb
servicePort: 8085
Minikube commands (this works without a problem):
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
kubectl apply -f traefik-ui.yaml
kubectl apply -f wws-ingress.yaml
And in Microk8s I tried:
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
microk8s.kubectl apply -f traefik-ui.yaml
microk8s.kubectl apply -f wws-ingress.yaml
After testing my setup on another machine and seeing that it is working there I found out that something is wrong with my machine and after spending a good amount of time on this with the help of two of my colleagues and trying everything we found out that the problem is related to iptable in my machine and we solved it as described here: https://github.com/ubuntu/microk8s/issues/72

Authentication Process is not triggered when using IBM App ID in IBM Cloud Kubernetes Service

I'm trying to use this feature: https://cloud.ibm.com/docs/services/appid?topic=appid-kube-auth#kube-auth
I've followed the steps in the documentation, but the authentication process is not triggered. Unfortunately I don't see any errors and don't know what else to do.
Here is my sample service (nginx.yaml):
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: nginx
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
ports:
- name: http
port: 80
protocol: TCP
selector:
app: nginx
type: NodePort
Here is my sample service (ingress.yaml). Replace 'niklas-heidloff-4' with your cluster name and 'niklas-heidloff-appid' with the name of your App ID service instance.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-with-app-id
annotations:
ingress.bluemix.net/appid-auth: "bindSecret=binding-niklas-heidloff-appid namespace=default requestType=web"
spec:
tls:
- hosts:
- niklas.niklas-heidloff-4.us-south.containers.appdomain.cloud
secretName: niklas-heidloff-4
rules:
- host: niklas.niklas-heidloff-4.us-south.containers.appdomain.cloud
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
Here are the steps to reproduce the sample:
First create a new cluster with at least two worker nodes in Dallas as described in the documentation. Note that it can take some extra time to get a public IP for your cluster.
Then create a App ID service instance.
Then invoke the following commands (replace 'niklas-heidloff-4' with your cluster name):
$ ibmcloud login -a https://api.ng.bluemix.net
$ ibmcloud ks region-set us-south
$ ibmcloud ks cluster-config niklas-heidloff-4 (and execute export....)
$ ibmcloud ks cluster-service-bind --cluster niklas-heidloff-4 --namespace default --service niklas-heidloff-appid
$ kubectl apply -f nginx.yaml
$ kubectl apply -f ingress.yaml
After this I could open 'https://niklas.niklas-heidloff-4.us-south.containers.appdomain.cloud/' but the authentication process is not triggered and the page opens without authentication.
I tried the steps mentioned in the link and this is how it worked for me.
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
ingress.bluemix.net/appid-auth: "bindSecret=binding-appid-ks namespace=default requestType=web serviceName=nginx idToken=false"
spec:
tls:
- hosts:
- test.vidya-think-cluster.us-south.containers.appdomain.cloud
secretName: vidya-think-cluster
rules:
- host: test.vidya-think-cluster.us-south.containers.appdomain.cloud
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
I added the following web redirect URL in the authentication settings of App ID service - http://test.vidya-think-cluster.us-south.containers.appdomain.cloud/appid_callback.
Now, when you try accessing the app at http://test.vidya-think-cluster.us-south.containers.appdomain.cloud/ you should see the redirection to App ID
Looks like idToken=false is a mandatory parameter as there is an error when you run kubectl describe myingress
Error: Failed to apply ingress.bluemix.net/appid-auth annotation. Error annotation format error : One of the mandatory fields not valid/missing for annotation ingress.bluemix.net/appid-auth