I am testing Istio 1.1, but the collection of metrics is not working correctly.
I can not find what the problem is. I followed this tutorial and I was able to verify all the steps without problems.
If I access prometheus I can see the log of some requests.
On the other hand, if I access Jaeger, I can not see any service (only 1 from Istio)
Grafana is also having some strange behavior, most of the graphs do not show data.
In istio 1.1, the default sampling rate is 1%, so you need to send at least 100 requests before the first trace is visible.
This can be configured through the pilot.traceSampling option.
Related
Kubernetes. Prometheus. A test application that can send metrics or a simple phrase to an HTTP request.
The end goal is to see graphs in Grafana. I want to make sure that everything works up to the current setup moment.
Now I want to see the metrics by URL, to see that the ServiceMonitor is working correctly, that they reach the Graphana. But so far I haven't been able to find a good enough troubleshooting guide.
I assume it's okay not to show everyone your metrics. But I want to make sure that this is normal behavior and which way to google in case I want to allow it.
This is a completely intended behavior when trying to use Grafana to visualize data from Managed Service for Prometheus. Metrics will be available via http request when trying to browse Grafana, as during the deployment we port-forward the frontend service to our local machine. As it is not a good practice to leave our metrics available to everyone for some security reasons, Grafana port-forwards the frontend service to our local machine.
Google Cloud APIs all require authentication using OAuth2. However, Grafana doesn't support OAuth2 authentication for Prometheus data sources. To use Grafana with Managed Service for Prometheus, you must use the Prometheus UI as an authentication proxy.
You can refer to these documentations to know more about Grafana & How-to-Deploy-Grafana.
Problem setting. Suppose I have 2 pods, A and B. I want to be able to dynamically scale pod A based on some arbitrary number from some arbitrary source. Suppose that pod B is such a source: for example, it can have an HTTP server with an endpoint which responds with the number of desired replicas of pod A at the moment of request. Or maybe it is an ES server or a SQL DB (does not matter).
Question. What kubernetes objects do I need to define to achieve this (apart from HPA)? What configuration should HPA have to know that it needs to look up B for current metric? How should API of B look like (or is there any constraints?)?
Research I have made. Unfortunately, the official documentation does not say much about it, apart from declaring that there is such a possibility. There are also two repositories, one with some go boilerplate code that I have trouble building and another one that has no usage instructions whatsoever (though allegedly does fulfil the "external metrics over HTTP" requirement).
By having a look at the .yaml configs in those repositories, I have reached a conclusion that apart from Deployment and Service one needs to define an APIService object that registers the external or custom metric in the kubernetes API and links it with a normal service (where you would have your pod) and a handful of ClusterRole and ClusterRoleBinding objects. But there is no explanation about it. Also I could not even list existing APIServices with kubectl in my local cluster (of 1.15 version) like other objects.
The easiest way will be to feed metrics into Prometheus (which is a commonly solved problem), and then setup a Prometheus-based HPA (also a commonly solved problem).
1. Feed own metrics to Prometheus
Start with Prometheus-Operator to get the cluster itself monitored, and get access to ServiceMonitor objects. ServiceMonitors are pointers to services in the cluster. They let your pod's /metrics endpoint be discovered and scraped by a prometheus server.
Write a pod that reads metrics from your 3rd party API and shows them in own /metrics endpoint. This will be the adapter between your API and Prometheus format. There are clients of course: https://github.com/prometheus/client_python#exporting
Write a Service of type ClusterIP that represents your pod.
Write a ServiceMonitor that points to a service.
Query your custom metrics thru Prometheus dashboard to ensure this stage is done.
2. Setup Prometheus-based HPA
Setup Prometheus-Adapter and follow the HPA walkthrough.
Or follow the guide https://github.com/stefanprodan/k8s-prom-hpa
This looks like a huge pile of work to get the HPA. However, only the adapter pod is a custom part here. Everything else is a standard stack setup in most of the clusters, and you will get many other use cases for it anyways.
We plan to use Istio on our AWS EKS K8s-Cluster and have explored Ingress, Egress and auth via Keycloak so far, but we are a little lost how we can easily aggregate all logs to a single, easy to query and monitoring place. Istio docs are just mentioning Mixer will be dead, but no really help what else could be done.
Scope: Access-Logs, Istiod logs and Apllication/Microservices logs from stdout as well.
mTLS is enabled cluster-wide (that seems to make problems using log sidecars)
We use Kiali, but that's not exactly what we need.
We are looking more on something like an ELK-Stack or Graylog but idealy more lightweight. We thought of Grafana Loki, but that its quite calm when you google for Istio+Loki... Seems to be not working.
So my question: What would be a best practice for log aggregation with Istio on K8s for all these logs in one place, and what is needed to getting it started (tutorial/how-to link?)
Thanks in advance!
Istio docs are just mentioning Mixer will be dead, but no really help what else could be done.
As mentioned in documentation
Mixer is deprecated. The functionality provided by Mixer is being moved into the Envoy proxies.
If you take a look at 1.5 release notes documentation
A new model for extensibility
Istio has long been the most extensible service mesh, with Mixer plugins allowing custom policy and telemetry support and Envoy extensions allowing data plane customization. In Istio 1.5 we’re announcing a new model that unifies Istio’s extensibility model with Envoy’s, using WebAssembly (Wasm). Wasm will give developers the ability to safely distribute and execute code in the Envoy proxy – to integrate with telemetry systems, policy systems, control routing and even transform the body of a message. It will be more flexible and more efficient, eliminating the need for running a Mixer component separately (which also simplifies deployments).
Read our Wasm blog post, and look out for posts from Google, Solo.io and the Envoy community for much more detail about this exciting work!
After mixer beeing deprecated there is something new, called telemetry v2
Telemetry V2 lacks a central component (Mixer) with access to K8s metadata, the proxies themselves require the metadata necessary to provide rich metrics. Additionally, features provided by Mixer had to be added to the Envoy proxies to replace the Mixer-based telemetry. Istio Telemetry V2 uses two custom Envoy plugins to achieve just that.
It´s well described there.
So it´s not like mixer is dead and there is nothing else to replace it.
What would be a best practice for log aggregation with Istio on K8s for all these logs in one place, and what is needed to getting it started (tutorial/how-to link?)
I would start with Remotely Accessing Telemetry Addons, which shows how to configure Istio to expose and access the telemetry addons(prometheus,grafana, kiali and jaeger/zipkin).
Everything depends on your use case, by default you can enable prometheus,grafana kiali and jaeger/zipkin versions provided by istio.
Additionally take a look at istio documentation metrics,logs and tracing.
We use Kiali, but that's not exactly what we need. We are looking more on something like an ELK-Stack or Graylog but idealy more lightweight. We thought of Grafana Loki, but that its quite calm when you google for Istio+Loki... Seems to be not working.
As far as I know you should be able to configure istio with elk, but it´s not easy and there is lack of documentation about that.
There is information what you have to do with elasticsearch to make it work and related github issue about that, so I assume elk would work too. Take a look at this tutorial.
I have set up container insights as described in the Documentation
Is there a way to remove some of the metrics sent over to CloudWatch ?
Details :
I have a small cluster ( 3 client facing namespaces, ~ 8 services per namespace ) with some custom monitoring, logging, etc in their own separate namespaces, and I just want to use CloudWatch for critical client facing metrics.
The problem I am having is that the Agent sends over 500 metrics to CloudWatch, where I am really only interested in a few of the important ones, especially as AWS bills per metric.
Is there any way to limit which metrics get sent to CloudWatch?
It would be especially helpful if I could only sent metrics from certain namespaces, for example, exclude the kube-system namespace
My configmap is:
cwagentconfig.json: |
{
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "*****",
"metrics_collection_interval": 60
}
},
"force_flush_interval": 5
}
}
I have searched for a while now, but clouldn't really find anything on:
"metrics_collected": {
"kubernetes": {
I've looked as best I can and you're right, there's little or nothing to find on this topic. Before I make the obvious-but-unhelpful suggestions of either using Prometheus or asking on the AWS forums, a quick look at what the CloudWatch agent actually does.
The Cloudwatch agent gets container metrics either from from cAdvisor, which runs as part of kubelet on each node, or from the kubernetes metrics-server API (which also gets it's metrics from kubelet and cAdvisor). cAdvisor is well documented, and it's likely that the Cloudwatch agent uses the Prometheus format metrics cAdvisor produces to construct it's own list of metrics.
That's just a guess though unfortunately, since the Cloudwatch agent doesn't seem to be open source. That also means it may be possible to just set a 'measurement' option within the kubernetes section and select metrics based on Prometheus metric names, but probably that's not supported. (if you do ask AWS, the Premium Support team should keep an eye on the forums, so you might get lucky and get an answer without paying for support)
So, if you can't cut down metrics created by Container Insights, what are your other options? Prometheus is easy to deploy, and you can set up recording rules to cut down on the number of metrics it actually saves. It doesn't push to Cloudwatch by default, but you can keep the metrics locally if you have some space on your node for it, or use a remote storage service like MetricFire (the company I work for, to be clear!) which provides Grafana to go along with it. You can also export metrics from Cloudwatch and use Prometheus as your single source of truth, but that means more storage on your cluster.
If you prefer to view your metrics in Cloudwatch, there are tools like Prometheus-to-cloudwatch which actually scrape Prometheus endpoints and send data to Cloudwatch, much like (I'm guessing) the Cloudwatch Agent does. This service actually has include and exclude settings for deciding which metrics are sent to Cloudwatch.
I've written a blog post on EKS Architecture and Monitoring in case that's of any help to you. Good luck, and let us know which option you go for!
I have a mule application which mostly does HTTP requests, which is logging as plain text. I want to push these logs as metrics to Prometheus. Since this is a legacy application it would take a substantial amount of time to change code and push metrics directly into Prometheus storage.
Idea is to show Prometheus metrics in Grafana Dashboard.
Is there any intermediate tool that converts plain text to metrics?
Anything that helps with this requirement.
FYI- We have Nagios and Splunk which is doing this task as of now, we are looking to move our solution to Prometheus and Grafana
In situations like these you can use tools like https://github.com/fstab/grok_exporter to convert logs into metrics.