We plan to use Istio on our AWS EKS K8s-Cluster and have explored Ingress, Egress and auth via Keycloak so far, but we are a little lost how we can easily aggregate all logs to a single, easy to query and monitoring place. Istio docs are just mentioning Mixer will be dead, but no really help what else could be done.
Scope: Access-Logs, Istiod logs and Apllication/Microservices logs from stdout as well.
mTLS is enabled cluster-wide (that seems to make problems using log sidecars)
We use Kiali, but that's not exactly what we need.
We are looking more on something like an ELK-Stack or Graylog but idealy more lightweight. We thought of Grafana Loki, but that its quite calm when you google for Istio+Loki... Seems to be not working.
So my question: What would be a best practice for log aggregation with Istio on K8s for all these logs in one place, and what is needed to getting it started (tutorial/how-to link?)
Thanks in advance!
Istio docs are just mentioning Mixer will be dead, but no really help what else could be done.
As mentioned in documentation
Mixer is deprecated. The functionality provided by Mixer is being moved into the Envoy proxies.
If you take a look at 1.5 release notes documentation
A new model for extensibility
Istio has long been the most extensible service mesh, with Mixer plugins allowing custom policy and telemetry support and Envoy extensions allowing data plane customization. In Istio 1.5 we’re announcing a new model that unifies Istio’s extensibility model with Envoy’s, using WebAssembly (Wasm). Wasm will give developers the ability to safely distribute and execute code in the Envoy proxy – to integrate with telemetry systems, policy systems, control routing and even transform the body of a message. It will be more flexible and more efficient, eliminating the need for running a Mixer component separately (which also simplifies deployments).
Read our Wasm blog post, and look out for posts from Google, Solo.io and the Envoy community for much more detail about this exciting work!
After mixer beeing deprecated there is something new, called telemetry v2
Telemetry V2 lacks a central component (Mixer) with access to K8s metadata, the proxies themselves require the metadata necessary to provide rich metrics. Additionally, features provided by Mixer had to be added to the Envoy proxies to replace the Mixer-based telemetry. Istio Telemetry V2 uses two custom Envoy plugins to achieve just that.
It´s well described there.
So it´s not like mixer is dead and there is nothing else to replace it.
What would be a best practice for log aggregation with Istio on K8s for all these logs in one place, and what is needed to getting it started (tutorial/how-to link?)
I would start with Remotely Accessing Telemetry Addons, which shows how to configure Istio to expose and access the telemetry addons(prometheus,grafana, kiali and jaeger/zipkin).
Everything depends on your use case, by default you can enable prometheus,grafana kiali and jaeger/zipkin versions provided by istio.
Additionally take a look at istio documentation metrics,logs and tracing.
We use Kiali, but that's not exactly what we need. We are looking more on something like an ELK-Stack or Graylog but idealy more lightweight. We thought of Grafana Loki, but that its quite calm when you google for Istio+Loki... Seems to be not working.
As far as I know you should be able to configure istio with elk, but it´s not easy and there is lack of documentation about that.
There is information what you have to do with elasticsearch to make it work and related github issue about that, so I assume elk would work too. Take a look at this tutorial.
Related
Kubernetes. Prometheus. A test application that can send metrics or a simple phrase to an HTTP request.
The end goal is to see graphs in Grafana. I want to make sure that everything works up to the current setup moment.
Now I want to see the metrics by URL, to see that the ServiceMonitor is working correctly, that they reach the Graphana. But so far I haven't been able to find a good enough troubleshooting guide.
I assume it's okay not to show everyone your metrics. But I want to make sure that this is normal behavior and which way to google in case I want to allow it.
This is a completely intended behavior when trying to use Grafana to visualize data from Managed Service for Prometheus. Metrics will be available via http request when trying to browse Grafana, as during the deployment we port-forward the frontend service to our local machine. As it is not a good practice to leave our metrics available to everyone for some security reasons, Grafana port-forwards the frontend service to our local machine.
Google Cloud APIs all require authentication using OAuth2. However, Grafana doesn't support OAuth2 authentication for Prometheus data sources. To use Grafana with Managed Service for Prometheus, you must use the Prometheus UI as an authentication proxy.
You can refer to these documentations to know more about Grafana & How-to-Deploy-Grafana.
In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like to instal Prometheus, Loki, and Grafana for metrics, logs, and visualisation respectively. Unfortunately, I've found no articles with instructions about using Prometheus without K8s.
But is it possible? If so, is it a good way? And how to do this? I also know that Prometheus & Loki can automatically detect services in the K8s to extract metrics and logs, but will the same work for a custom orchestration system?
Can't comment about Loki, but Prometheus is definitely doable.
Prometheus supports a number of service discovery mechanisms, k8s being just on of them. If you look at the list of options (the ones ending with _sd_config) you can see if your provider is there.
If it is not then a generic service discovery can be used. Maybe DNS-based discovery will work with your custom system? If not then with some glue code a file based service discovery will almost certainly work.
Yes, I'm running Prometheus, Loki etc. just fine in a AWS ECS cluster. It just requires a bit more configuration especially regarding service discovery (if you are not already using something like ECS Service Disovery or Hashicorp Consul)
The Openshift documentation is absolutely abysmal. I can't find direct documentation for any of the objects that are available.
I did find a section in the Kubernetes docs that seems to describe the ability to do something like this...
https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
But it wasn't super clear how this translates to OoenShift, or how to actually implement this IPVS mode for a service.
Answered on Slack, but short version it is not an option for this user given their situation.
For others, IPVS does support this but it is enabled and configured at a global level. A better option is usually a userspace proxy, often via the Ingress system.
I have 4 microservices running on my laptop listening at various ports. Can I use Istio to create a service mesh on my laptop so the services can communicate with each other through Istio? All the links on google about Istio include kubernetes but I want to run Istio without Kubernetes. Thanks for reading.
In practice, not really as of this writing, since pretty much all the Istio runbooks and guides are available for Kubernetes.
In theory, yes. Istio components are designed to be 'platform independent'. Quote from the docs:
While Istio is platform independent, using it with Kubernetes (or infrastructure) network policies, the benefits are even greater, including the ability to secure pod-to-pod or service-to-service communication at the network and application layers.
But unless you know really well the details of each of the components: Envoy, Mixer, Pilot, Citadel, and Galley and you are willing to spend a lot of time it becomes not practically feasible to get it running outside of Kubernetes.
If you want to use something less tied to Kubernetes you can take a look at Consul, although it doesn't have all the functionality Istio has, it has overlap with some of its features.
I do some googles, and found that istio claim to support apps running outside k8s, like in vm. But I never try.
https://istio.io/latest/news/releases/0.x/announcing-0.2/#cross-environment-support
https://jimmysong.io/blog/istio-vm-odysssey/
Is there any way to get statistics such as service / endpoint access for services defined in Kubernetes cluster?
I've read about Heapster, but it doesn't seem to provide these statistics. Plus, the whole setup is tremendously complicated and relies on a ton of third-party components. I'd really like something much, much simpler than that.
I've been looking into what may be available in kube-system namespace, and there's a bunch of containers and services, there, Heapster including, but they are effectively inaccessible because they require authentication I cannot provide, and kubectl doesn't seem to have any API to access them (or does it?).
Heapster is the agent that collects data, but then you need a monitoring agent to interpret these data. On GCP, for example, that's fluentd who gets these metrics and sends to Stackdriver.
Prometheus is an excellent monitoring tool. I would recommend this one, if youare not on GCP.
If you would be on GCP, then as mentioned above you have Stackdriver Monitoring, that is configured by default for K8s clusters. All you have to do is to create a Stackdriver accound (this is done by one click from GCP Console), and you are good to go.