I'm running the following helm chart (https://github.com/helm/charts/tree/master/stable/prometheus) with server.replicaCount =2 and server.statefulSet.enabled = true.
For storage i use two Manage Disks (not Azure Files that is not POSIX) (2 PV and 2 PVC) are created during the deployment of the chart.
My question is:
Is this an HA solution? Are the metrics written to both prometheus instances (a service with a public ip and and headless "service" are created) and replicated to both disks?
How this replicas really work?
Thanks,
Sadly, as Piotr noted, this is not a true HA offering and Thanos is generally the preferred way to go for this kind of setup, but not without it's own gotchas. The amount of clusters you have is a factor, and you might need some sort of tooling account to be able to follow changes all the way through.
What I can offer you is this excellent talk, which includes a live demo and shows how this works in practice.
No, this is not HA solution. This only scales the deployment to have 2 replicas at all times which both are on statefulsets.
In order to achieve HA monitoring on Kuberetes there needs to be dynamic failure detection and routing tools involved.
There are couple of articles about getting prometheus work with HA:
Deploying an HA Prometheus in Kubernetes on AWS — Multiple Availability Zone Gotchas
HA Kubernetes Monitoring using Prometheus and Thanos
The number of replicas only instructs deployment to always have at least 2 running instances of the deployment pods. You can find more information about replicas in Kubernetes documentation.
In the helm chart documentation, there seems to be other options like server.service.statefulsetReplica.enabled and server.service.statefulsetReplica.replica but I think those are just tools that can help to create HA prometheus. Not a ready from the get go solution.
Hope it helps.
Related
I am new to Kubernetes monitoring and self-healing. I wonder what kind of self-healing Kubernetes can provide, such as restart failed pod if necessary? anything else? what Kubernetes cannot provide.
As for Kubernetes monitoring, what kind of metrics we need to monitor in order to operate on Kubernetes instead of Kubernetes self-healing?
Any ideas welcomed. Thanks.
I'm afraid your question goes beyond what is possible to answer here on stackoverflow.
Yes, k8s is able to restart/reschedule pods. If you are already a bit familiar with key concepts, maybe pod-lifecycle is a point to start.
If you have little knowledge about k8s basics, I suggest you study Deployments, DaemonSets, Services etc. because Monitoring in k8s relies heavily on them!
You did not say what kind of metrics you are interested in.
For system metrics like io/cpu time etc. you can start with e.g. Kubernetes Metrics Server.
If you want to get insights into k8s metrics (how many services, uptime, etc.) have a look at kube-state-metrics which is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
Have fun with k8s
Cheers
Configure the liveness and readiness probes for pod health. And the Restart policy along with it. You can do more with services and replica sets.
I tried to launch Istio on Google Kubernetes Engine using the Google Cloud Deployment Manager as described in the Istio Quick Start Guide.
My goal is to have a cluster as small as possible for a few very lightweight microservices.
Unfortunately, Istio pods in the cluster failed to boot up correctly when using a
1 node GKE
g1-small or
n1-standard-1
cluster.
For example, istio-pilot fails and the status is "0 of 1 updated replicas available - Unschedulable".
I did not find any hints that the resources of my cluster are exceeded so I am wondering:
What is the minimum GKE cluster size to successfully run Istio (and a few lightweight microservices)?
What I found is the issue Istio#216 but it did not contain the answer. Also, of course, the cluster size depends on the microservices but I am basically interested in the minimum cluster to start with.
As per this page
If you use GKE, please ensure your cluster has at least 4 standard GKE nodes. If you use Minikube, please ensure you have at least 4GB RAM.
I cannot find any articles answering question: Is it safe/right to deploy Spinnaker to same Kubernetes cluster which Spinnaker will manage? Mainly I mean for production, HA deployments.
I think the architectures of Spinnaker and Kubernetes compliment each other very well, and running Spinnaker in the same K8s cluster it is managing is definitely safe.
As per your comment in #mdirkse's answer, there is a codelab, which is official Spinnaker documentation, that explains how to create a set of basic pipelines for deploying code from a Github repo to a production Kubernetes cluster in the form of a Docker container.
In this documentation, it specifically states the following:
We will be deploying Spinnaker to the same Kubernetes cluster it will be managing. ...
Not sure if this is exactly what you are looking for though.
I'm not sure about "right", but I'd definitely say that it is safe to run Spinnaker on the same Kubernetes cluster that it manages, if you set it up right. Kubernetes (and Docker) gives you all the tools you need to properly separate Spinnaker from the other things running on the cluster (namespaces, quotas, node affinities etc). Indeed the whole point of Kubernetes is to be able to easily run software in an HA/fault tolerant way, and since Spinnaker consists of a collection of stateless microservices it really plays to the strenghts of k8s.
We have some Kubernetes clusters that have been deployed using kops in AWS.
We really like using the upstream/official images.
We have been wondering whether or not there was a good way to monitor the systems without installing software directly on the hosts? Are there docker containers that can extract the information from the host? I think that we are likely concerned with:
Disk space (this seems to be passed through to docker via df
Host CPU utilization
Host memory utilization
Is this host/node level information already available through heapster?
Not really a question about kops, but a question about operating Kubernetes. kops stops at the point of having a functional k8s cluster. You have networking, DNS, and nodes have joined the cluster. From there your world is your oyster.
There are many different options for monitoring with k8s. If you are a small team I usually recommend offloading monitoring and logging to a provider.
If you are a larger team or have more specific needs then you can look at such options as Prometheus and others. Poke around in the https://github.com/kubernetes/charts repository, as I know there is a Prometheus chart there.
As with any deployment of any form of infrastructure you are going to need Logging, Monitoring, and Metrics. Also, do not forget to monitor the monitoring ;)
I am using https://prometheus.io/, it goes naturally with kubernetes.
Kubernetes api already exposes a bunch of metrics in prometheus format,
https://github.com/kubernetes/ingress-nginx also exposes prometheus metrics (enable-vts-status: "true"), and you can also install https://github.com/prometheus/node_exporter as a daemonset to monitor CPU, disk, etc...
I install one prometheus inside the cluster to monitor internal metrics and one outside the cluster to monitor LBs and URLs.
Both send alerts to the same https://github.com/prometheus/alertmanager that MUST be outside the cluster.
It took me about a week to configure everything properly.
It was worth it.
I'm currently learning about Kubernetes and still trying to figure it out. I get the general use of it but I think that there still plenty of things I'm missing, here's one of them. If I want to run Kubernetes on my public cloud, like GCE or AWS, will Kubernetes spin up new VMs by itself in order to make more compute for new pods that might be needed? Or will it only use a certain amount of VMs that were pre-configured as the compute pool. I heard Brendan say, in his talk in CoreOS fest, that Kubernetes sees the VMs as a "sea of compute" and the user doesn't have to worry about which VM is running which pod - I'm interested to know where that pool of compute comes from, is it configured when setting up Kubernetes? Or will it scale by itself and create new machines as needed?
I hope I managed to be coherent.
Thanks!
Kubernetes supports scaling, but not auto-scaling. The addition and removal of new pods (VMs) in a Kubernetes cluster is performed by replication controllers. The size of a replication controller can be changed by updating the replicas field. This can be performed in a couple ways:
Using kubectl, you can use the scale command.
Using the Kubernetes API, you can update your config with a new value in the replicas field.
Kubernetes has been designed for auto-scaling to be handled by an external auto-scaler. This is discussed in responsibilities of the replication controller in the Kubernetes docs.