I have a pod that has following chart:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
// ...
paths:
- /api/myservice
My pod exposes api and additionally a /prometheus endpoint that is accessible through /api/myservice/prometheus.
I would like to have prometheus visible inside my cluster but not from /api/myservice/prometheus. How I can achieve that?
You can add an Ingress rule that redirects the endpoint to the default-backend:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: block
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- myapp.to
rules:
- host: myapp.to
http:
paths:
- path: /metrics
backend:
serviceName: ingress-default-backend
servicePort: 8080
If you just need internal access, I don't think you should put it into ingress. You could make use of the DNS Service together with Kubernetes cluster already.
If your prometheus service is running the same namespace as your working pod, you could use following address to access it.
http://prometheus-svc-name
If your prometheus service is running in a different namespace, you could use:
http://prometheus-svc-name.prometheus-namespace.svc.cluster.local
to access the service.
Update:
More clear about the question after comments.
You could write another rule to hide your /prometheus endpoint like this:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
// ...
paths:
- /api/myservice => myservice
- /api/myservice/prometheus => 404 default backend.
nginx-ingress will match the longest route first. It will route /api/myservice/prometheus to 404. See this documentation
Best way to achieve this would be to remove the prometheus endpoint from ingress and if you are aware prometheus supports auto discovery using annotations on the service, configure prometheus for k8s auto discovery and then add annotations in you service yaml file.
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
change the endpoint to point to your API.
/api/myservice/my-api or use app-root annotation, to point to your api inside /api/myservice:
nginx.ingress.kubernetes.io/app-root: /path/to/my/app/root/dir
more details here: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite
Related
I created an ingress service from the kubernetes documentation, but I found that not putting below annotation made service unavailable. Why was this so?
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: "false"
Main reason to ask this question was that in K8s documentation, I found below code for ingress, but this did not work until and unless I put the above annotation. So, why does the below code not work?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-wear-watch
namespace: app-space
spec:
rules:
- http:
paths:
- path: /wear
pathType: Prefix
backend:
service:
name: wear-service
port:
number: 8080
- path: /watch
pathType: Prefix
backend:
service:
name: video-service
port:
number: 8080
Based on k8s docs, all you need to know in this specific case is that
you can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.
[annotations are] Directives from the end-user to the implementations to modify behavior or engage non-standard features.
So basically this specific annotations are used to modify behaviour of ingress and are specififc to nginx ingress controller (this means that if you use different controller, this annotations wont work).
All supported annotations for nginx ingress controler are explained in Nginx Ingress Controller documention.
I have deployed helm chart for prometheus-operator and currently all the services are up and running.
I have already a ingress controller at place and it is routing our internal services perfectly.
But now I am creating ingress for Prometheus, Grafana & AlertManager and it is not working as expected.
My Prometheus Ingress YAML -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: prom
namespace: monitoring
spec:
rules:
- host: abc.xyz.com
http:
paths:
- backend:
serviceName: my-prom-check-prometheus-o-prometheus
servicePort: 9090
path: /prometheus
Output is - 404 not found
But for Grafana it doesn't give 404 and rather gives Invalo key error -
Kindly advise on it. I haven't modified values.yaml and directly installed it.
Prometheus is not aware of what you are trying to achieve and that's why it's redirecting to unknown destination, you get 404 page not found error message because the http request that gets to the target webserver is claiming for the content of /prometheus directory instead of / from which it is actually served.
As far as I know you have to tell prometheus to accept traffic on the new path. I assume if you change path from /prometheus to / it should work.
If you want it on /prometheus you would either have to properly configure the prometheus operator chart values, I assume they are externalUrl and routePrefix.
prometheus.prometheusSpec.externalUrl -> The external URL the Prometheus instances will be available under. This is necessary to generate correct URLs. This is necessary if Prometheus is not served from root of a DNS name.
prometheus.prometheusSpec.routePrefix -> The route prefix Prometheus registers HTTP handlers for. This is useful, if using ExternalURL and a proxy is rewriting HTTP routes of a request, and the actual ExternalURL is still true, but the server serves requests under a different route prefix. For example for use with kubectl proxy. Default value is /
There is related github issue about that
OR
Configure your nginx to rewrite from /prometheus to /.
The following ingress definition should resolve your problem
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- backend:
serviceName: my-prom-check-prometheus-o-prometheus
servicePort: 9090
path: /prometheus(/|$)(.*)
Rewrite that was used above will ensure that the original access path /prometheus gets rewritten to / before reaching the target Pod.
There is tutorial about that.
If an ingress controller is already in place, prometheus can be accessed on the custom domain by using following values in your values.yaml.
Is there a specific reason you are manually trying to configure and deploy prometheus ingress yourself?
Using following values in your values.yaml will create a prometheus ingress for you.
ingress:
enabled: true
annotations: {kubernetes.io/ingress.class: "nginx"}
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - prometheus.domain.com
hosts: []
## Paths to use for ingress rules - one path should match the prometheusSpec.routePr
##
paths:
- /
Now, you'll be able to access your prometheus UI on localhost via browser.
I am trying to setup Prometheus in k8 cluster, able to run using helm. Accessing dashboard when i expose prometheus-server as LoadBalancer service using external ip.
Same does not work when I try to configure this service as ClusterIP and making it as backend using ingress controller. Receiving 404 error, any thoughts on how to troubleshoot this?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
with above ingress definition in place, url “http://<>/prometheus/ getting redirected to http://<>/graph/ and then 404 error page getting rendered. When url adjusted to http://<>/prometheus/graph some of webcontrols gets rendered with lots of errors on browser console.
Prometheus might be expecting to have control over the root path (/).
Please change the Ingress to prometheus.example.com and it should work fine. (Changing it to a subdomain)
Please change your Ingress configuration file, add host field:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: prometheus.example.com
http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
then apply changes executing command:
$ kubectl aply -f your_ingress_congifguration_file.yaml
The host header field in a request provides the host and port
information from the target URI, enabling the origin server to
distinguish among resources while servicing requests for multiple
host names on a single IP address.
Please take a look here: hosts-header.
Ingress definition: ingress.
Useful information: helm-prometheus.
Useful documentation: ingress-path-matching.
So I am in the process of migrating my bare metal cluster onto GKE and ran into an issue with the ingress. On my bare metal cluster, I used the ingress controller from nginxinc which worked fine. Below is a sample of an Ingress file of a particular deployment:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: mynamespace
name: app-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: myhost
http:
paths:
- path: /dev/appname(/|$)(.*)
backend:
serviceName: app
servicePort: 80
Basically, when a user types in myhost/dev/appname, myhost is resolved to a HAProxy server. The proxy then routes the request to the NodePort that the ingress service is running on.
I tried to do the same thing on my GKE cluster with the only exception being that the Ingress controller on the GKE cluster is exposed using a LoadBalancer as per the documentation
However I get a 502 error if I try to access the URL.
TL;DR: Looking for the best way to access various applications (deployments) deployed on a GKE cluster using URL's like: myhost/dev/firstapp and myhost/dev/secondapp etc.
You can use Kong Ingress as your ingress controller on GKE for your path based ingresses. You can install Kong Ingress from GCP Marketplace. It is easy to integrate and also supports various plugins for authenticating, monitoring etc.
You'll get detailed information and installation instructions from https://github.com/Kong/google-marketplace-kong-app#basic-usage
I would follow this guide on setting up Nginx on GKE. The ingress looks like below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-resource
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /hello
backend:
serviceName: hello-app
servicePort: 8080
You should be able to access your app externally http://external-ip-of-ingress-controller/hello
Now to debug 502 issue verify that the health-checks from the Loadbalancer to your app is passing or not.
I created a service and each service is creating a new load balancer, I don't want to create a new load balancer for each service. For that, I found solution ingress controller but it's not happening.
I will try to describe the objects you need in just words.
You don't need to create a load balancer for each service. When you're using an ingress controller (like nginx), the ingress controller itself will be the type load balancer. All your other services need to be something like ClusterIP type.
Afterwards you can decide how to link your ClusterIP services with the Nginx LoadBalancer: create an ingress for each service or one ingress that exposes each service based on some rule (like paths as #harsh-manvar shows in the post above).
When you say "it's not happening", it would be good if you could provide details on your setup.
In order for Nginx ingress controller to work, it needs to be defined either as a NodePort or LoadBalancer service type. The examples provided in the nginx documentation are using LoadBalancer. However, LoadBalancer only works when your cluster supports this object (that means running in most cloud providers like AWS/GCP/Azure/DigitalOcean or newer versions of minikube). On the other hand, NodePort will expose the ingress controller on the Kubernetes node where it runs (when using minikube, that usually means a VM of sorts which then needs to be port forwarded to be accessible).
To use ingress in a local environment, you can look into minikube. All you need is to run minikube addons enable ingress and it will deploy an nginx controller for you. Afterwards, all you need to do is define an ingress and depending on your setup you may need to use kubectl port-forward to port forward port 80 on an nginx controller pod to a local port on your machine.
There are different types of services: ClusterIP, NodePort, LoadBalancer and ExternalName. You can specify it in spec.type. Actually the default one, when not specified is not LoadBalancer, but ClusterIP, so in your case, simply leave away the type: LoadBalancer definition and use your serviceName as backend in your ingress resource. Example:
spec:
rules:
- host: your.fully.qualified.host.name
http:
paths:
- backend:
serviceName: your-internal-service-name
servicePort: 80
path: /
Keep in mind that for some cloud providers there's also the possibility to use an internal LoadBalancer without a public IP. This is done by adding an annotation to the service configuration. For Azure AKS it looks like this:
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
For Google's GKE the annotation is cloud.google.com/load-balancer-type: "Internal"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: "nginx"
certmanager.k8s.io/cluster-issuer: wordpress-prod
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- test.test.com
secretName: prod
rules:
- host: test.test.com
http:
paths:
- path: /service-1
backend:
serviceName: service-1
servicePort: 80
- path: /service-2
backend:
serviceName: service-2
servicePort: 5000
Sharing here documentation for ingress to target multiple services you can redirect to multi-service.
Using this you can access services like
https://test.test.com/service-1
https://test.test.com/service-2
Following documentation you should do the following.
More information: kubernetes.github.com
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: rewrite
namespace: default
spec:
rules:
- host: rewrite.bar.com
http:
paths:
- backend:
serviceName: http-svc
servicePort: 80
path: /something(/|$)(.*)
For example, the ingress definition above will result in the following rewrites:
rewrite.bar.com/something rewrites to rewrite.bar.com/
rewrite.bar.com/something/ rewrites to rewrite.bar.com/
rewrite.bar.com/something/new rewrites to rewrite.bar.com/new