We want to expose the metrics of our Hibernate caches into Prometheus, and have for the time being built our own metrics for the caches, but since Infinispan 10 provides native metrics support, we'd rather use that.
So when I curl the localhost:9990/metrics and look for infinispan related metrics, I find nothing. I do find jgroups, and our own metrics.
The configuration for the metrics in the standalone.xml is:
<subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0"
security-enabled="false"
exposed-subsystems="*"
prefix="${wildfly.metrics.prefix:wildfly}"
/>
We've also added "statistics-enabled=true" to the defined infinispan cache-containers:
<cache-container name="hibernate"
default-cache="local-query"
module="org.infinispan.hibernate-cache"
statistics-enabled="true">
I've searched the web for Infinispan, Wildfly, metrics, but I only find generic metrics articles about how you can create your own, or the announcements of added support for metrics in Infinispan.
According to the subsystem configuration all metrics should be exposed. Is there anything that we need to configure in addition to enable infinispan metrics inside wildfly?
I had the same issue and found out that there is a bug in Wildfly 20, so that infinispan statistics aren't exported. See WFLY-14063 and the fixing pull-request.
The fix version mentioned in the ticket is 22.0.0.Beta1.
Not sure if it is going to work, there is a metrics tag in cache-container that needs to be configured/enabled:
<cache-container statistics="true">
<metrics gauges="true" histograms="true" />
</cache-container>
See the infinispan configuration doc
Related
I am looking for a java agent to monitor the metrics of all JVM running processes in my POD. I want to collect the metrics for each process like CPU, Memory and forward them to stdout or Splunk. How Can I achieve that?
There are a bunch of ways to do this, but I'll highlight 3 of them.
Jolokia - Jolokia is remote JMX with JSON over HTTP. You essentially enable this in your java arguments and it will spin up a server that allows you to request json. You can also install hawtio online for a gui in kubernetes. If you go to Red Hat, the JBoss image has Jolokia installed by defaults. That's how it does it's health checks.
Prometheus - The Prometheus agent can also be installed. You can also ask for metrics on an http/https port just like jolokia.
OpenTelemetry - Opentelemetry is newer and I honestly haven't played around with it yet.
If you run Red Hat JBoss images , both Prometheus and Jolokia extensions are added by default. Hope this helps, Jay
I would like to monitor all ELK service running in our kubernetes clusters to be sure, that is still running properly.
I am able to monitor Kibana portal via URL. ElasticSearch via Prometheus and his metrics (ES have some interested metrics to be sure, that ES is working well).
But exist something similar for Filebeat, Logstash, ... ? Have these daemons some exposed metrics for Prometheus, which is possible to watching and analizing it states?
Thank you very much for all hints.
There is an exporter for ElasticSearch found here: https://github.com/prometheus-community/elasticsearch_exporter and an exporter for Kibana found here: https://github.com/pjhampton/kibana-prometheus-exporter These will enable your Prometheus to scrape the endpoints and collect metrics.
We are also working on a new profiler inside of OpenSearch which will provide much more detailed metrics and fix a lot of bugs. That will also natively provide an exporter for Prometheus to scrape : https://github.com/opensearch-project/OpenSearch/issues/539 you can follow along here, this is in active development if you are looking for an open-source alternative to ElasticSearch and Kibana.
Yes, both the beats and logstash have metrics endpoint for monitoring.
These monitoring endpoints are built to be consumed using metricbeat, but since they return a json you can use other tools to monitor it.
For logstash the metrics endpoint is enabled by default, listening on localhost at port 9600, and from the documentation you have these two endpoints:
node
node_stats
For the beats family you need to enable it as if you would consume the metrics using metricbeat, this documentation explains how to do that.
Then you will have two endpoints:
stats
state
So you would just need to use those endpoints to collect the metrics.
Infinispan isn't replicating caches across the cluster; the user is asked to login again when Keycloak on any node in the cluster isn't healthy. Can you please help point out what we are doing wrong?
We followed the documentation for Keycloak Infinispan setup (https://www.keycloak.org/docs/latest/server_installation/#sticky-sessions) and set the owners for all caches to 3. Below is an example excerpt of our cache owner setup (domain/configuration/domain.xml)
<subsystem xmlns="urn:jboss:domain:infinispan:11.0">
<cache-container name="keycloak">
<distributed-cache name="sessions" owners="3"/>
...
Details about our Keycloak setup
Domain clustered mode
1 primary, 2 secondary
Hosted on AWS and ALB has sticky session enabled.
Any help is appreciated.
In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like to instal Prometheus, Loki, and Grafana for metrics, logs, and visualisation respectively. Unfortunately, I've found no articles with instructions about using Prometheus without K8s.
But is it possible? If so, is it a good way? And how to do this? I also know that Prometheus & Loki can automatically detect services in the K8s to extract metrics and logs, but will the same work for a custom orchestration system?
Can't comment about Loki, but Prometheus is definitely doable.
Prometheus supports a number of service discovery mechanisms, k8s being just on of them. If you look at the list of options (the ones ending with _sd_config) you can see if your provider is there.
If it is not then a generic service discovery can be used. Maybe DNS-based discovery will work with your custom system? If not then with some glue code a file based service discovery will almost certainly work.
Yes, I'm running Prometheus, Loki etc. just fine in a AWS ECS cluster. It just requires a bit more configuration especially regarding service discovery (if you are not already using something like ECS Service Disovery or Hashicorp Consul)
Based on this link, we can setup a prometheus rule to monitor mm2:
https://github.com/apache/kafka/tree/trunk/connect/mirror#monitoring-an-mm2-process
Tried to apply this rule to a configmap on kubernetes using the confluent connect distribution, but no success neither any error.
Could be deleted or not available those jmx rules on the confluent connect distribution? (we are using the 5.5.0)
Confluent has nothing to do with the issue since MM2 is Apache Licensed.
You'll need to configure your prometheus jmx-exporter with the correct MBeans
I suggest using VisualVM or JMXTerm to view what info is available