Here is the case, i want to visualize a query result of data taken periodically into dashboard (like grafana or kibana), the problem is I don't know which technology stack to use, should I use ELK stack, Prometheus + Grafana, or using Tableau, the requirements are:
First, it has support for multiple (hundreds) of database server as data source, currently I use Postgresql.
Second, it has support for running one query to all database instance and collect the result into one centralized server to be then displayed on dashboard.
Third, it has support for period/schedule set up (cron-like scheduler) for managing how often data should be queried from all database servers.
Fourth, it has support for alerting/notification system, where i can use existing platform library without much code needed.
Fifth, it has to be opensource project, with good reputation and quite large community support.
Thanks
You can achieve your objective with ELK Stack. In Kibana, you will see basic Dashboard. If you want more detail Dashboard view, then you can integrate Elastic Search with Grafana as well using Lucene query.Below links will help you:
https://www.elastic.co/blog/logstash-jdbc-input-plugin
https://discuss.elastic.co/t/how-can-i-schedule-logstash-every-second-for-jdbc-input-plugin/27393/11
https://grafana.com/blog/2016/03/09/how-to-effectively-use-the-elasticsearch-data-source-in-grafana-and-solutions-to-common-pitfalls/#lucene
Related
So I know the JDBC Rivers plugin is deprecated so even though it is being used I'd ideally not want to look at using something that is no longer supported.
However I have a few tables in a Postgres database with values that I need to be able to search in a Kibana view. Im new to the ELK stack but i've been messing around with some of their samples to get familiar.
I've seen some mentions of using Stored Procedures/Triggers from Postgres to send to Logstash. Although im not sure if this is the best way. Im not a developer but a QA so my coding skills are "ok" as im used to writing automation tests/etc...
What would be the best way to do this? I would want to probably capture updates to these tables (probably new inserts or updates) OR be able to poll the data every X period of time (30s or something). Lets pretend it's for a weather station and the tables contain humidity data from different weather sensors.
I'd want to be able to search in a Kibana view the Values/Station ID/etc...
Is this doable? Is there maybe a better way than using Triggers/Stored procedures?
I ended up using the JDBC driver and following https://www.elastic.co/blog/logstash-jdbc-input-plugin to get it moving and working (Which it does move). But it was a lot of setup for anyone that may see this answer.
I am trying to know about GRAFANA advantages over other tools.Can anybody list what are all the metrics that GRAFANA Displays/Collects ? This helps a lot to starters.
Thanks in Advance!!!
You seem to be a little confused about what Grafana does. It is a visualization tool, and is agnostic about the types of data you're visualizing. That is what makes it much more powerful than tools that are more tightly integrated with a particular collector/database.
As long as you can get the data you want to visualize (and in 4.x alert on) into one of the many time-series databases that Grafana supports, you'll be able to visualize that data on Grafana dashboards.
If you want to get an idea of what time-series databases are supported, you can look at the docs for Graphite and the other built-in datasources http://docs.grafana.org/datasources/graphite/ (see the others in the side menu), and find more datasources supported via plugins on https://grafana.net/plugins
I need to some visualize data from a Postgresql in Kibana. I have also ElasticSearch installed just in case. So how visualize data from a Postgresql in Kibana? Of course, I don't need the whole database, but only data returned by a custom sql query.
Also, I want it to be as simple as possible, I wouldn't like to use libraries I really don't need to use.
Kibana was built with Elastisearch in mind.
Having used it quite a lot in a startup I worked for, I can tell you that even the front-end query DSL (built on Lucene) will only work with Elasticsearch (or might need some serious tweaks).
I would advise you to push your data into Elasticsearch, and just work with Kibana the way it was made for :)
I have a postgres database with a large number of time series metrics
Various operators are interested in different information and I want to provide an interface where they can chart the data, make comparisons and optionally export data as a csv.
The two solutions I have come across so far are, graphite and grafana, but both these solutions tie you down to storage engines and none support postgres.
What I am looking for is an interface similar to grafana, but which allows me to hook up any backend I want. Are there any tools out there, similar to grafana, which allow you to hook up any backend you want (or even just postgres).
Note that the data I am collecting is highly sensitive, and is required by other areas of the application and so is not suitable for storing in graphite.
The other alternative I see would be to setup a trigger on the postgres DB to feed data into graphite as it arrives, but again, not ideal.
You may want to replace Graphite's storage backend with postgresql. Here is a good primer.
2018 update: Grafana now supports PostgreSQL (link).
What I am looking for is an interface similar to grafana, but which allows me to hook up any backend I want
Thats possible with grafana . Check this guide which shows how to create a datasource plugin for a datasource thats currently not supported.
We would like to use OrientDB Graph in an Azure environment. Does anybody has experience using it? We also would like to know if high availability from OrientDB is required under Azure cloud? Azure already offers high availability for Azure storage, Azure Drive and SQL. I understand that they have replications and load balancing built in.
This is super important because we prefer not to get into the business of replications and infrastructure management.
Thanks
So you can spin up 2 or more machines and install OrientDB on them, then configure them together as a distributed cluster. However I haven't been able to find any way that is simpler, easier to do. I am interested in this topic too.
Azure does have features such as geo-replication, which is protects your data against a major data-center incident but doesn't provide any performance benefit and will not make it highly available.
Although pretty reliable, occasionally Microsoft will reboot servers for updates, so to protect against downtime you can use affinity groups so that, of your 2 or more servers, one will always be online. This however does need to be used in conjunction with database replication and ideally load balancing.
It's also worth noting that OrientDB recommends clusters have an odd number of servers as this can prevent conflicts when synchronising data after a communication issue between the servers.
I am using it in amazon and I had to create a java project to monitor http requests inserts and queries. The queries are very fast but takes longer inserting data .
I recommend this type of graph database mode to decrease the time of the queries. Also if you have empty fields OrientDB manages very well compared to other databases .
If you need help with the java project can response to this post and I´ll help u.
I hope it helps. Good luck.