I'm trying to get my head around how to get prometheus https://hub.helm.sh/charts/stable/prometheus collect etcd stats. I understand I need to set tls for it, but have a hard time to find good way to do it without manual additional ansible steps. Is there the way I can get etcd certs on worker node and mount them to prometheus pod?
Following the Monitoring External Etcd Cluster With Prometheus Operator you can easily configure Prometheus to scrape metrics from ETCD.
We can do all of that by creating certs as kubernetes secrets and
adding a tlsConfig to our service monitor. Let me walk you through the
whole process.
The steps are:
1) Create etcd service
2) Create/attach endpoints for etcd service
3) Create service monitor with appropriate tlsconfig. below example
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: etcd
name: etcd
namespace: kube-system
spec:
endpoints:
- interval: 30s
port: metrics
scheme: https
tlsConfig:
caFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt
certFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.crt
keyFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.key
serverName: etcd-cluster
jobLabel: k8s-app
selector:
matchLabels:
k8s-app: etcd
4) Create Etcd Client Certificates
5) Create Kubernetes Secrets along with previously created certificate and key for prometheus and etcd ca. This will allow prometheus to securely connect to etcd.
Example:
kubectl -n monitoring create secret kube-etcd-client-certs --from-file=etcd-client-ca.crt=etcd-client.ca.crt --from-file=etcd-client.crt=etcd-client.crt --from-file=etcd-client.key=etcd-client.key
6) Update prometheus.yaml to include there names of the created secrets.
7) delploy etcd-service,servicemonitor and prometheus manifests to cluster
kubectl apply -f etcd-service.yaml
kubectl apply -f etcd-serviceMon.yaml
kubectl apply -f prometheus-prometheus.yaml
Enjoy
Related
I deployed Prometheus with an Helm chart from Rancher. Targets such as Alertmanager, Prometheus, Grafana, Node-exporter, Kubelet etc. are configured automatically. The endpoint from alertmanager refers to the IP address of the specific pod for example. I also configured multiple targets successfully like Jira and Confluence.
Since the service external-dns is running in the namespace kube-system, it's also configured automatically. But only this service is getting the error Context deadline exceeded.
I checked in a random pod if those metrics are accessible by running the command curl -s http://<IP-ADDRESS-POD>:7979/metrics. Also did this with the service ip address (kubectl get service external-dns and curl-s http://<IP-ADDRESS-SVC>:7979/metrics).
Both of these curl commands returned the metrics within a second. So increasing the scrape timeout won't help.
But when I exec in the Prometheus container and use the promtool debug metrics command it shows the same behaviour like in my browser. The external-dns returns a timeout with both of the IP addresses and if I try this with another target it just returns the metrics.
I also don't think it's a SSL issue, because I already injected the correct CA bundle for the targets Jira and Confluence.
So anybody an idea? :)
I had to edit the NetworkPolicy in the kube-system namespace. The containers from the cattle-monitoring-system namespace are now allowed to access the containers from the kube-system namespace. You can upload your NetworkPolicies here and it visualizes which resources has access or not. The NetworkPolicy looks like this now:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-network-policy
namespace: kube-system
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
name: cattle-monitoring-system
- from:
- podSelector: {}
podSelector: {}
policyTypes:
- Ingress
I would like to plot in Grafana, the metrics for the readiness/liveness probes for some of my pods. Currently, the way I am deploying prometheus in my cluster is using:
helm install prometheus stable/prometheus -n prometheus
I am able to see all standard metrics by going to the prometheus UI, but I am trying to figure out how to get the probes metrics. Apparently the kubelet expose these metrics in /metrics/probes, but I don't know how to configure them. Moreover, I noted that apparently the "standard" metrics are grabbed from the kubernetes api-server on the /metrics/ path, but so far I haven't configured any path nor any config file (I just run the above command to install prometheus). I am assuming that this /metrics/ path is hardcoded somewhere in the helm chart repo, but since I want to get the metrics for the kubelets, this might be trickier, as my understanding is that he api-server lives in the master k8s node, and the kubelet only runs on the worker nodes (so I have no idea where to point the /metrics/probes path).
Use Prometheus Operator and create ServiceMonitor in which you can specify the endpoints, ports exposed by kubelet or any other component. Prometheus will start scraping the endpoints for metrics.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: kubelet
labels:
k8s-app: kubelet
spec:
jobLabel: k8s-app
endpoints:
- port: https-metrics
scheme: https
interval: 30s
tlsConfig:
insecureSkipVerify: true
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
- port: https-metrics
scheme: https
path: /metrics/cadvisor
interval: 30s
honorLabels: true
tlsConfig:
insecureSkipVerify: true
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
selector:
matchLabels:
k8s-app: kubelet
namespaceSelector:
matchNames:
- kube-system
I have created a k8 cluster in GKE.
I have a docker registry created in Artifactory, this artifactory is hosted on AWS. I have a route53 entry for docker-repo.aws.abc.com in aws.abc.com Hosted zone in AWS
Now, I need to configure my cluster so that the docker images are pulled from artifactory.
I went through documentation and understand I will have to add stubDomain in my kube-dns configmaps.
kubectl edit cm kube-dns -n kube-system
apiVersion: v1
data:
stubDomains: |
{"aws.abc.com" : ["XX.XX.XX.XX"]}
kind: ConfigMap
metadata:
creationTimestamp: 2019-05-21T14:30:15Z
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: kube-dns
namespace: kube-system
resourceVersion: "7669"
selfLink: /api/v1/namespaces/kube-system/configmaps/kube-dns
uid: f378aa5f-7bd4-11e9-9df2-42010aa93d03
However, still docker pull command fails.
docker pull docker-repo.aws.abc.com/abc-sampleapp-java/abc-service:V-57bc9c9-201
Error response from daemon: Get https://docker-repo.aws.abc.com/v2/: dial tcp: lookup docker-dev-repo.aws.abc.com on 169.254.169.254:53: no such host
Note: When I make an entry in /etc/hosts file on worker nodes, docker pull works fine.
Pulling an image from registry on pod start uses different DNS settings than when we call DNS from pods inside a cluster.
When Kubernetes starts new pod, it schedules it to the node and then agent on the node named kubelet calls container engine (Docker) to download an image and run it with designed configuration.
Docker uses system DNS to resolve the address of a registry, because it works right on our host system, not in the Kubernetes, that is why any DNS settings will not affect DNS resolving on the image downloading stage. https://github.com/kubernetes/kubernetes/issues/8735 is a discussion about it on Github.
If we want to change DNS settings and override the registry IP to use it on image downloading stage, we should set it in the host system. In the configuration, we need to modify DNS settings on all your nodes in the cluster. The simplest way to do it is using /etc/hosts file and adding a record with your custom IP, e.g. 192.168.1.124 example.com.
After that modifications, Docker on nodes will use the record from /etc/hosts for your registry instead of global DNS records, because that file has higher priority and you will be able to run pods with your image.
To update the host file.
you can use a DeamonSet with Security Context as privileged see below:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hosts-fix-script
namespace: kube-system
labels:
app: hosts-fix-script
spec:
selector:
matchLabels:
name: hosts-fix
template:
metadata:
labels:
name: hosts-fix
spec:
hostPID: true
containers:
- name: hosts-fix-script
image: gcr.io/google-containers/startup-script:v1
imagePullPolicy: Always
securityContext:
privileged: true
env:
- name: STARTUP_SCRIPT
value: |
#!/bin/bash
echo "10.0.0.11 onprem.registry" >> /etc/hosts
echo 'Done'
you need to run the kubectl apply -f
I would like to expose my Kubernetes Managed Digital Ocean (single node) cluster's service on port 80 without the use of Digital Ocean's load balancer. Is this possible? How would I do this?
This is essentially a hobby project (I am beginning with Kubernetes) and just want to keep the cost very low.
You can deploy an Ingress configured to use the host network and port 80/443.
DO's firewall for your cluster doesn't have 80/443 inbound open by default.
If you edit the auto-created firewall the rules will eventually reset themselves. The solution is to create a separate firewall also pointing at the same Kubernetes worker nodes:
$ doctl compute firewall create \
--inbound-rules="protocol:tcp,ports:80,address:0.0.0.0/0,address:::/0 protocol:tcp,ports:443,address:0.0.0.0/0,address:::/0" \
--tag-names=k8s:CLUSTER_UUID \
--name=k8s-extra-mycluster
(Get the CLUSTER_UUID value from the dashboard or the ID column from doctl kubernetes cluster list)
Create the nginx ingress using the host network. I've included the helm chart config below, but you could do it via the direct install process too.
EDIT: The Helm chart in the above link has been DEPRECATED, Therefore the correct way of installing the chart would be(as per the new docs) is :
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
After this repo is added & updated
# For Helm 2
$ helm install stable/nginx-ingress --name=myingress -f myingress.values.yml
# For Helm 3
$ helm install myingress stable/nginx-ingress -f myingress.values.yml
#EDIT: The New way to install in helm 3
helm install myingress ingress-nginx/ingress-nginx -f myingress.values.yaml
myingress.values.yml for the chart:
---
controller:
kind: DaemonSet
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
daemonset:
useHostPort: true
service:
type: ClusterIP
rbac:
create: true
you should be able to access the cluster on :80 and :443 via any worker node IP and it'll route traffic to your ingress.
since node IPs can & do change, look at deploying external-dns to manage DNS entries to point to your worker nodes. Again, using the helm chart and assuming your DNS domain is hosted by DigitalOcean (though any supported DNS provider will work):
# For Helm 2
$ helm install --name=mydns -f mydns.values.yml stable/external-dns
# For Helm 3
$ helm install mydns stable/external-dns -f mydns.values.yml
mydns.values.yml for the chart:
---
provider: digitalocean
digitalocean:
# create the API token at https://cloud.digitalocean.com/account/api/tokens
# needs read + write
apiToken: "DIGITALOCEAN_API_TOKEN"
domainFilters:
# domains you want external-dns to be able to edit
- example.com
rbac:
create: true
create a Kubernetes Ingress resource to route requests to an existing Kubernetes service:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: testing123-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: testing123.example.com # the domain you want associated
http:
paths:
- path: /
backend:
serviceName: testing123-service # existing service
servicePort: 8000 # existing service port
after a minute or so you should see the DNS records appear and be resolvable:
$ dig testing123.example.com # should return worker IP address
$ curl -v http://testing123.example.com # should send the request through the Ingress to your backend service
(Edit: editing the automatically created firewall rules eventually breaks, add a separate firewall instead).
A NodePort Service can do what you want. Something like this:
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
type: NodePort
selector:
app: MyApp
ports:
- protocol: TCP
nodePort: 80
targetPort: 80
This will redirect incoming traffic from port 80 of the node to port 80 of your pod. Publish the node IP in DNS and you're set.
In general exposing a service to the outside world like this is a very, very bad idea, because the single node passing through all traffic to the service is both going to receive unbalanced load and be a single point of failure. That consideration doesn't apply to a single-node cluster, though, so with the caveat that LoadBalancer and Ingress are the fault-tolerant ways to do what you're looking for, NodePort is best for this extremely specific case.
I have asked myself this question and invested time researching it. Running out of time. Can someone point me in the right direction?
I have created a kubernetes cluster on minikube, with its Ingress, Services and Deployments. There is a whole configuration of services in there.
Can, now, I point this kubectl command to another provider like VMWareFusion, AWS , Azure, not to forget Google Cloud.
I know about kops. My understanding is that although this is the design goal of kops but presently it only supports AWS.
Yes, you can use different clusters via the context. List them using kubectl config get-contexts and switch between them using kubectl config use-context.
I would like to suggest you couple of things the way i worked out with kubernetes, From my local system to production my environment remains consistent.
I use kubeadm to create a kubernetes cluster on my local machine. And I maintain all my kubernetes resources like Services, Pods, Deployment etc.. in a yaml as my deployment files.
All my services and pods are saved in a yaml file e.g. counter.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: deployment-counter
namespace: default
labels:
module: log-counter
spec:
replicas: 1
selector:
matchLabels:
module: log-counter
template:
metadata:
labels:
module: log-counter
spec:
containers:
- name: container-counter
image: busybox
command:
- "/bin/sh"
- "-c"
- 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
tolerations:
- key: ud_application
operator: Equal
value: docxtract
effect: NoSchedule
- key: ud_module
operator: Exists
effect: NoSchedule
strategy:
type: RollingUpdate
On my local kubernetes cluster provisioned by kubeadm I deploy it as follow
kubectl apply -f counter.yaml
And on Production i have a kubernetes cluster provisioned by kubeadm too and i deploy it the same way.
kubectl apply -f counter.yaml
PS:
kubeadm is a tool provided by kubernetes to provision a kubernetes cluster.