I am looking for a way to decouple Prometheus from applications by putting a Kafka in between to achieve something like this:
+-------------+ +----------+ +--------------+
| Application +--metrics--->+ Kafka +------>+ Prometheus |
+-------------+ +----------+ +--------------+
In order to solve this problem I have two questions:
Are there any Java libraries that abstract metrics representation so my app will not depend on Prometheus in any ways?
Are there any reliable Kafka reporter?
Any comments or suggestions are welcome.
The Prometheus Java client library is designed so that you can used it with other monitoring systems, and indeed many open source and commercial monitoring systems do so as the Prometheus text format is becoming a defacto-standard.
Prometheus is a pull based system and it is not at all recommended to try and convert it to push, you're making your life harder for no real reason. It is recommended to have Prometheus scrape the application directly.
Are there any Java libraries that abstract metrics representation so my app will not depend on Prometheus in any ways?
Yes - StatsD (I've used Datadog's dogstatsd), which is push-based.
Related - Which StatsD client should I use for a java/grails project?
Then you can use StatsD as a relay to Prometheus - https://github.com/prometheus/statsd_exporter
Or use something else that supports remote reads
Related
Does anyone know whether PostgreSQL has built-in /metrics (or something like that)?
I've searched through the web and all I found was third party open source tools that send metrics to Prometheus
Thanks :)
Unfortunately, PostgreSQL doesn't have any /metrics endpoint. All sorts of metrics can be obtained through SQL queries from system tables.
Here is the list of monitoring tools.
For Prometheus, there is a pretty good exporter.
Problem -> I want to deploy JanusGraph as separate service on Kubernetes. which storage backend should i use for cassandra. Is it CQL or cassandrathrift?? Cassandra is running as stateful service on Kubernetes.
Detailed Description-> As per JanusGraph doc, in case of Remote Server Mode, storage backend should be cql.
JanusGraph graph = JanusGraphFactory.build().
set("storage.backend", "cql").
set("storage.hostname", "77.77.77.77").
open();
Even they mentioned that Thrift is deprecated going ahead with Cassandra 2.1 & I am using Cassandra 3.
But in some blog, they have mentioned that rest api call from JanusGraph to Cassandra is possible only through Thrift.
Is Thrift really required? Can't we use CQL as storage backend for rest api call as well?
Yes, you absolutely should use the cql storage backend.
Thrift is deprecated, disabled by default in the current version of Cassandra (version 3), and has been removed from Cassandra version 4.
I would also be interested in reading the blog post you referenced. Are you talking about IBM's Rest API mentioned in their JanusGraph-utils Git repo? That confuses me as well, because I see both Thrift and CQL config happening there. In any case, I would go with the cql settings and give it a shot.
tl;dr;
Avoid Thrift at all costs!
I have a use case in which metrics will be written to kafka topics and from there I have to send these metrics to a grafana collection point.
Can it be done without a datasource?
Any idea how it can be done?
You need to store your metrics somewhere and then visualize it. If you want to use Grafana, you can store metric data from Kafka to Elasticsearch via connectors. I think you can also store them in InfluxDB, Graphite, and Prometheus. You can use data source plugins that Grafana provides.
Also using Kibana is a good option. Kibana is like Graphana. Elasticsearch and Kibana are part of Elastic Stack.
Refer to the below pics.
1 :
2 :
I found this open source code that is basically a kafka plugin for Grafana.
https://github.com/zylklab/lorca
You can either use it straightaway or get inspired to write your own Grafana plugin.
I am investigating options for monitoring our installation in Swisscom's cloud-foundry. My objectives are the following:
monitor performance indicators for deployed application (such as cpu, disk, memory)
monitor performance indicators for services (slow queries, number of queries, ideally also some metrics on hitting quotas)
So far, I understand the options are the following (including some BUTs):
I used a very nice TOP cf-plugin (github)
This works very well. It seems that it registers itself to get the required firehose nozzles and consume data.
That is very useful for tracing / ad-hoc monitoring, but not very good for a serious infrastructure monitoring.
Another way I found is to use firehose-syslog solution.
This can be deployed as an app to (as far as I understand) do the job in similar way, as the TOP cf plugin.
The problem is, that it requires registered client, so it can authenticate with the doppler endpoint. For some reason, the top-cf-plugin does that automatically / in another way.
Last option i am considering is to build the monitoring itself to the App (using a special buildpack)
That can be for example done with Datadog. But it seems to also require a dedicated uaa client to register the Nozzle.
I would like to check, if somebody is (was) on the similar road, has some findings.
Eventually I would like to raise the following questions towards the swisscom community support:
is it possible to register uaac client to be able to ingest events through the firehose nozzle from external service? (this requires admin credentials if I was reading correctly)
is there an alternative way to authenticate with the nozzle (for example using a special user and his authentication token?)
is there any alternative to monitor the CF deployments in Swisscom? Eventually, is there a paper, blogpost or other form of documentation, that would be helpful in this respect (also for other users of AppCloud)?
Since it requires admin permissions, we can not give out UAA clients for the firehose.
However, there are different ways to get metrics in context of a user.
CF API
You can obtain basic metrics of a specific app by polling the CF API:
https://apidocs.cloudfoundry.org/5.0.0/apps/get_detailed_stats_for_a_started_app.html
However, since you have to poll (and for each app), it's not the recommended way.
Metrics in syslog drain
CF allows devs to forward their logs to syslog drains; in more recent versions, CF also sends metrics to this syslog drain (see https://docs.cloudfoundry.org/devguide/deploy-apps/streaming-logs.html#container-metrics).
For example, you could use Swisscom's Elasticsearch service to store these metrics and then analyze it using Kibana.
Metrics using loggregator (firehose)
The firehose allows streaming logs to clients for two types of roles:
Streaming all logs to admins (which requires a UAA client with admin permissions) and streaming app logs and metrics to devs with permissions in the app's space. This is also what the cf logs command uses. cf top also works this way (it enumerates all apps and streams the logs of each app).
However, you will find out that most open source tools that leverage the firehose only work in admin mode, since they're written for the platform operator.
Of course you also have the possibility to monitor your app by instrumenting it (white box approach), for example by configuring Spring actuator in a Spring boot app or by including an agent of your favourite APM vendor (Dynatrace, AppDynamics, ...)
I guess this is the most common approach; we've seen a lot of teams having success by instrumenting their applications. Especially since advanced monitoring anyway requires you to create your own metrics as the firehose provided cpu/memory metrics are not that powerful in a microservice world.
However, option 2. would be worth a try as well, especially since the ELK's stack metric support is getting better and better.
I'm writing a microservice for an existing Ignite cluster. I need to have basic communications with Ignite Messaging system, and don't need other Ignite capabilities. I don't want to include Ignite libraries as it will bloat my microservice - ignite.zip is about 10 times larger than my server and I only need a small subset of functionality.
How can I send messages to existing Ignite cluster and receive messages from it?
EDIT: Ignite documentation lists REST API as one of ways to use Ignite. I'm not sure how it can be used to work with Ignite messaging - suppose I want to receive message as soon as it becomes available in the Ignite messaging? I don't want to poll for messages, as that's not efficient enough for me. If using REST API, the question becomes: how (if it's possible) to receive message using Ignite REST API from the distributed messaging system?
To do this you need only one JAR - ignite-core, which doesn't have any additional dependencies.
To achieve functionality, you can start a client node in your application and use IgniteMessaging API: https://apacheignite.readme.io/docs/messaging