In my kubernetes cluster I would like to do monitoring so I installed grafana.
I would like to access the grafana dashboard as http://example.com/monitoring, so I tried to include this in my ingress configuration
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: example.com
http:
paths:
- path: /monitoring/(.*)
backend:
serviceName: grafana
servicePort: 80
The idea is to add other paths there as well, for example / for the website.
I noticed that Grafana redirects http://example.com/monitoring to http://example.com/login. Of course this should have been http://example.com/monitoring/login. What would be the preferred way to fix this. Can it be done with ingress or should I somehow tell Grafana that it is behind a /monitoring path (if possible)?
I've installed grafana using this using Helm.
UPDATE: I've modified as suggested below the grafana chart's file values.yaml as follows
grafana.ini:
server:
domain: example.com
root_url: http://example.com/monitoring/
Now I get:
And the heml command I use to install grafana:
$> helm install stable/grafana -f values.yaml --set persistence.enabled=true --set persistence.accessModes={ReadWriteOnce} --set persistence.size=8Gi -n grafana
This is a common problem with services that are behind an HTTP reverse-proxy. Luckily, Grafana offers a way to let it know the context path it is running behind.
In grafana.ini (which is most possibly supplied via a ConfigMap to its Kubernetes deployment), you need to specify the variables like the following:
[server]
domain = example.com
root_url = http://example.com/monitoring/
See the full documentation here: https://grafana.com/docs/installation/behind_proxy/
Related
I am trying to deploy a SpringbBoot Java application hosted on Apache Tomcat on a Kubernetes cluster using Ngninx Ingress for URL routing. More specifically, I am deploying on Minikube on my local machine, exposing the SpringBoot application as a Cluster IP Service, and executing the
minikube tunnel
command to expose services to my local machine. Visually, the process is the following...
Browser -> Ingress -> Service -> Pod hosting docker container of Apache Tomcat Server housing SpringBoot Java API
The backend service requires a header "SM_USER", which for the moment can be any value. When running the backend application as a Docker container with port forwarding, accessing the backend API works great. However, when deploying to a Kubernetes cluster behind an Nginx ingress, I get 403 errors from the API stating that the SM_USER header is missing. I suspect the following. My guess is that the header is included with the request to the ingress, but removed when being routed to the backend service.
My setup is the following.
Deploying on Minikube
minikube start
minikube addons enable ingress
eval $(minikube docker-env)
docker build . -t api -f Extras/DockerfileAPI
docker build . -t ui -f Extras/DockerfileUI
kubectl apply -f Extras/deployments-api.yml
kubectl apply -f Extras/deployments-ui.yml
kubectl expose deployment ui --type=ClusterIP --port=8080
kubectl expose deployment api --type=ClusterIP --port=8080
kubectl apply -f Extras/ingress.yml
kubectl apply -f Extras/ingress-config.yml
edit /etc/hosts file to resolve mydomain.com to localhost
minikube tunnel
Ingress YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
# Tried forcing the header by manual addition, no luck (tried with and without)
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header SM_USER $http_Admin;
spec:
rules:
- host: mydomain.com
http:
paths:
- path: /ui
pathType: Prefix
backend:
service:
name: ui
port:
number: 8080
- path: /api
pathType: Prefix
backend:
service:
name: api
port:
number: 8080
Ingress Config YAML
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: default
labels:
app.kubernetes.io/name: my-ingress
app.kubernetes.io/part-of: my-ingress
data:
# Tried allowing underscores and also using proxy protocol (tried with and without)
enable-underscores-in-headers: "true"
use-proxy-protocol: "true"
When navigating to mydomain.com/api, I expect to receive the ROOT API interface but instead receive the 403 error page indicating that the SM_USER is missing. Note, this is not a 403 forbidden error regarding the ingress or access from outside the cluster, the specific SpringBoot error page I receive is from within my application and custom to indicate that the header is missing. In other words, my routing is definitely correct, and I am able to access the API, its just that the header is missing.
Are there configs or parameters I am missing? Possibly an annotation?
This is resolved. The issue was that the config was being applied in the Application namespace. Note, even if the Ingress object is in the application namespace, if you are using minikubes built in ingress functionality, the ConfigMap must be applied in the Nginx ingress namespace.
I am having the following issue.
I am new to GCP/Cloud, I have created a cluster in GKE and deployed our application there, installed nginx as a POD in the cluster, our company has a authorized SSL certificate which i have uploaded in Certificates in GCP.
In the DNS Service, i have created an A record which matched the IP of Ingress.
When i call the URL in the browser, it still shows that the website is still unsecure with message "Kubernetes Ingress controller fake certificate".
I used the following guide https://cloud.google.com/load-balancing/docs/ssl-certificates/self-managed-certs#console_1
however i am not able to execute step 3 "Associate an SSL certificate with a target proxy", because it asks "URL Maps" and i am not able to find it in the GCP Console.
Has anybody gone through the same issue like me or if anybody helps me out, it would be great.
Thanks and regards,
I was able to fix this problem by adding an extra argument to the ingress-nginx-controller deployment.
For context: my TLS secret was at the default namespace and was named letsencrypt-secret-prod, so I wanted to add this as the default SSL certificate for the Nginx controller.
My first solution was to edit the deployment.yaml of the Nginx controller and add at the end of the containers[0].args list the following line:
- '--default-ssl-certificate=default/letsencrypt-secret-prod'
Which made that section of the yaml look like this:
containers:
- name: controller
image: >-
k8s.gcr.io/ingress-nginx/controller:v1.2.0-beta.0#sha256:92115f5062568ebbcd450cd2cf9bffdef8df9fc61e7d5868ba8a7c9d773e0961
args:
- /nginx-ingress-controller
- '--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller'
- '--election-id=ingress-controller-leader'
- '--controller-class=k8s.io/ingress-nginx'
- '--ingress-class=nginx'
- '--configmap=$(POD_NAMESPACE)/ingress-nginx-controller'
- '--validating-webhook=:8443'
- '--validating-webhook-certificate=/usr/local/certificates/cert'
- '--validating-webhook-key=/usr/local/certificates/key'
- '--default-ssl-certificate=default/letsencrypt-secret-prod'
But I was using the helm chart: ingress-nginx/ingress-nginx, so I wanted this config to be in the values.yaml file of that chart so that I could upgrade it later if necessary.
So reading the values file I replaced the attribute: controller.extraArgs, which looked like this:
extraArgs: {}
For this:
extraArgs:
default-ssl-certificate: default/letsencrypt-secret-prod
This restarted the deployment with the argument in the correct place.
Now I can use ingresses without specifying the tls.secretName for each of them, which is awesome.
Here's an example ingress that is working for me with HTTPS:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: some-ingress-name
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
rules:
- http:
paths:
- path: /some-prefix
pathType: Prefix
backend:
service:
name: some-service-name
port:
number: 80
You can save your SSL/TLS certificate into the K8s secret and attach it to the ingress.
you need to config the TLS block in ingress, dont forget to add ingress.class details in ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tls-example-ingress
spec:
rules:
- host: mydomain.com
http:
paths:
-
backend:
serviceName: my-service
servicePort: 80
path: /
tls:
- hosts:
- mydomain.com
secretName: my-tls-secret
You can read more at : https://medium.com/avmconsulting-blog/how-to-secure-applications-on-kubernetes-ssl-tls-certificates-8f7f5751d788
You might be seeing something like this in browser :
that's from the ingress controller and wrong certificate attached to ingress or ingress controller default fake cert.
in my fault, I upgraded new tsl on cattle-system name space, but not in my name-space, therefore some how, ingress recognize with K8s fake cert.
Solution: upgrade all old cert to new cert (only ingress user cert, WARNING - system cert maybe damaged your system - cannot explain)
Also don't forget to check that Subject Alternative Name actually contains the same value the CN contains. If it does not, the certificate is not valid because the industry moves away from CN. Just learned this now
I am trying to deploy the kube-prometheus-stack.
I have added it as a dependency in the Chart.yaml as below.
...
dependencies:
- name: kube-prometheus-stack
version: 13.4.1
repository: https://prometheus-community.github.io/helm-charts
...
I have also configured an ingress rule to route the /grafana/?(.*) path to the service solutions-helm-grafana at port 80.
- path: /grafana/?(.*)
pathType: Prefix
backend:
service:
name: helm-grafana
port:
number: 80
However, when I try open /grafana/ in the browser, it returns a 404 after redirecting to /login. What templates do I need to add to successfully deploy ? Are there any examples that I can refer to ?
Hi #Moses can you try removing the ?(.*) from the path?
404 comes up when the ingress is not registered with the ingress controller.
Probably because the release has not been deployed successfully.
Try the following steps to debug the issue:-
Check if the pod have been deployed using kubectl get pods.
Try debugging the ingress object. kubectl describe ing <ing_object_name>.
Check if the Endpoints have been created using kubectl get ep.
Next,get the service endpoint using kubectl get service.
Use a busybox pod to curl and check whether the Grafana is being served via the service above.
Update:-
Add the following configuration to serve Grafana on a subpath
env:
GF_SERVER_DOMAIN: <domain>
GF_SERVER_ROOT_URL: https://<domain>/grafana/
GF_SERVER_SERVE_FROM_SUB_PATH: true
and use this path in ingress:
path: /grafana/
Sources:
Run Grafana behind reverse-proxy
Grafana confiugration root_url
I want to know if there is versioning for ingress config similar to what we have in deployments. Suppose there is a misconfiguration I would like to revert to the previous config.
I would like to understand about generation in ingress YAML config.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-match: 'new-nginx: header("foo", /^bar$/)' #Canary release rule. In this example, the request header is used.
nginx.ingress.kubernetes.io/service-weight: 'new-nginx: 50,old-nginx: 50' #The route weight.
creationTimestamp: null
generation: 1
name: nginx-ingress
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nginx-ingress
spec:
rules: ##The Ingress rule.
- host: foo.bar.com
http:
paths:
- backend:
serviceName: new-nginx
servicePort: 80
path: /
- backend:
serviceName: old-nginx
servicePort: 80
path: /
Kubernetes does not offer this natively, and neither does a management tool like Rancher.
If you want to do this, you need an infra-as-code tool, like Terreform, ansible, etc. The config files for these can be versioned in a repo.
Even without those, you can independently export a give ingress yaml, and commit it to a repo.
Slightly different way to look at a solution - you could use gitOps. I mean that you could put all your yaml's in a git repo, install ArgoCD on your cluster and then simply let ArgoCD do the syncing for you. The moment you realised you've messed something up in a yaml file, just revert the commit in the git repo. That way you maintain history and get a graceful non-opinionated solution.
I have a Kubernetes cluster deployed on AWS (EKS). I deployed the cluster using the “eksctl” command line tool. I’m trying to deploy a Dash python app on the cluster without success. The default port for Dash is 8050. For the deployment I used the following resources:
pod
service (ClusterIP type)
ingress
You can check the resource configuration files below:
pod-configuration-file.yml
kind: Pod
apiVersion: v1
metadata:
name: dashboard-app
labels:
app: dashboard
spec:
containers:
- name: dashboard
image: my_image_from_ecr
ports:
- containerPort: 8050
service-configuration-file.yml
kind: Service
apiVersion: v1
metadata:
name: dashboard-service
spec:
selector:
app: dashboard
ports:
- port: 8050 # exposed port
targetPort: 8050
ingress-configuration-file.yml (host based routing)
kind: Ingress
metadata:
name: dashboard-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: dashboard.my_domain
http:
paths:
- backend:
serviceName: dashboard-service
servicePort: 8050
path: /
I followed the steps below:
kubectl apply -f pod-configuration-file.yml
kubectl apply -f service-configuration-file.yml
kubectl apply -f ingress-confguration-file.yml
I also noticed that the pod deployment works as expected:
kubectl logs my_pod:
and the output is:
Dash is running on http://127.0.0.1:8050/
Warning: This is a development server. Do not use app.run_server
in production, use a production WSGI server like gunicorn instead.
* Serving Flask app "annotation_analysis" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
You can see from the ingress configuration file that I want to do host based routing using my domain. For this to work, I have also deployed an nginx-ingress. I have also created an “A” record set using Route53
that maps the “dashboard.my_domain” to the nginx-ingress:
kubectl get ingress
and the output is:
NAME HOSTS ADDRESS. PORTS. AGE
dashboard-ingress dashboard.my_domain nginx-ingress.elb.aws-region.amazonaws.com 80 93s
Moreover,
kubectl describe ingress dashboard-ingress
and the output is:
Name: dashboard-ingress
Namespace: default
Address: nginx-ingress.elb.aws-region.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
host.my-domain
/ dashboard-service:8050 (192.168.36.42:8050)
Annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: false
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: false
Events: <none>
Unfortunately, when I try to access the Dash app on the browser, I get a
502 Bad Gateway error from the nginx. Could you please help me because my Kubernetes knowledge is limited.
Thanks in advance.
It had nothing to do with Kubernetes or AWS settings. I had to change my python Dash code from:
if __name__ == "__main__":
app.run_server(debug=True)
to:
if __name__ == "__main__":
app.run_server(host='0.0.0.0',debug=True).
The addition of host='0.0.0.0' did the trick!
I think you'll need to check whether any other service is exposed at path / on the same host.
Secondly, try removing rewrite-target annotation. Also can you please update your question with output of kubectl describe ingress <ingress_Name>
I would also suggest you to use backend-protocol annotation with value as HTTP (default value, you can avoid using this if dashboard application is not SSL Configured, and only this application is served at the said host.) But, you may need to add this if multiple applications are served at this host, and create one Ingress with backend-protocol: HTTP for non SSL services, and another with backend-protocol: HTTPS to serve traffic to SSL enabled services.
For more information on backend-protocol annotation, kindly refer this link.
I have often faced this issue in my Ingress Setup and these steps have helped me resolve it.