I'm using the elk stack for the first time and I can import data with logstash but how do I link my mongodb to elastic instead?
Also, what is the best way to import bulk data?
I'm using the MEAN stack and the newest version of elk 5+. I am not using beats like filebeats but am willing to use if needed.
First, if you're using logstash successfully, then you don't need filebeats. (Although filebeats is much better than logstash).
I think you're confused about the other terms.. You don't hook up mongodb to elastic. When using the ELK stack, logstash is intended to send your logs to elasticsearch, and kibana is the UI layer to view your data.
If you really want to use mongodb (although I don't recommend), then you're using mongodb instead of elasticsearch.
If you are after searching MongoDB data in elasticsearch you will need to import it (from Mongo to Elasticsearch)
There are a few ways, one of them is described here: https://stackoverflow.com/a/24119066/6079633 - but i don't think it supports elastic 5
And there is this one https://github.com/ozlerhakan/mongolastic - which is according to elasticsearch website: "A tool that clones data from ElasticSearch to MongoDB and vice versa"
I know this answer could be late, but it may help other people.
If you need a tool to transfer your data from MongoDB to Elasticsearch, have a look into this mongoose plugin https://github.com/mongoosastic/mongoosastic/tree/master/docs, it's a great tool for automatically indexing MongoDB models into elasticsearch.
and you can transfer your collection data through indexing an existing collection in MongoDB
Related
I have lots of data in my MongoDB accross collection and I'd like to vizualise them, in graph-node way, or in relation represenation manner.
I know, that Mongo by default, isn't 'that good' solution for this case, and I should use something like Neo4j or ElasticSearch with Kibana instead. And also I know, that Mongo Compass has in-build charts.
And according to some articles on Mongo site and $graphLookup aggregation operator Mongo can be used in that way.
So my question is about a tool, which is capable to vizualise (or connect by values) entities between each other.
Probably there is some way to integrate and connect Kibana to MongoDB remote instance (not Atlas) or Graphana allow to solve this problem (but I haven't head much about it) so I'll be welcome to ay of your advice about it.
I have a scala microservice that serves as database api, and the database I am using is mongodb.
I want to add elasticsearch that will contain all the data that my mongodb have, and I need to keep it in sync when the mongodb is updated, how can I achieve it?
what would be the best approach to do this? is there some plugins or something that can help me with this task?
Look at the 5 Different ways to synchronize data from MongoDB to ElasticSearch, personally, I did it with Logstash where I simply filtered one collection and dumped to ES every 24 hrs, the use case is key to determine what strategy/tool is to use.
I'm looking for the right way so use ElasticSearch with MongoDB. I want to save several informations in MongoDB. Additionally i want to save a larger text with ElasticSearch to support complex fulltext-search.
My problem at the moment is:
I'm not sure what the best solution is for this. Most solutions i found to synchronize MongoDB with ElasticSearch are using "river" which is deprecated!
What is the best way to combine these two technologies?
Is it even the best way to save it in MongoDB and ElasticSearch?
I found multiple articles that explained, that ElasticSearch alone is not safe enough and that you have to use another DBMS.
Also under robustness on the mongoDB website I found this:
Unfortunately, Elasticsearch (and the components it's made of) does not currently handle OutOfMemory-errors very well.
[source]
So saving the data redundant is probably the best way.
Thanks in advance!
Hei,
We are also working with both Elasticsearch and MongoDb. We started with a river and after having a lot of issues with it we got rid of it before becoming deprecated. The way we do it is: when saving data to mongo we create a message in a queue which notifies the search storage to do the insert/delete operation with the given data.
So basically we keep them in sync manually and there will always be a delay between mongo and elaticsearch. The good part is that if elasticsearch would fail, we have implemented an endpoint which reimports the data from mongo to ES. Also, the structure inside ES it's different from the one in mongo. Before, it was a lot more complicated to do this with the river. Imagine that we even had our own custom implementation.
Hope my answer helps at least a bit.
Is it possible to use kibana front-end along with a mongodb back-end without using elastic search?
I'm using logstash to parse logs and store in mongodb and want to use kibana to display data?
If not, are there any alternatives to implement kibana+mongodb?
I'm afraid that Kibana is specifically designed to use the Elasticsearch API.
While they do both provide JSON responses, they don't return compatible data structures and even if they did, Mongo would not provide the same features (facets/filters) that Kibana makes heavy use of.
You could probably index your MongoDB data in Elasticsearch following instructions similar to https://coderwall.com/p/sy1qcw but then you are unnecessarily duplicating your data in 2 systems.
I am working on a project where we have millions of entries stored in MongoDB database and, i want to index all this data using SOLR.
After extensive Searching i came to know there are no proper "Data Import Handlers" for mongoDB database.
Can anyone tell me what are the proper approaches for indexing data in MongoDB using SOLR ?
I want to use all the features of SOLR and want it to be scalable in real-time. I saw one or two approaches from different posts but not sure how they will work real time..
Many Thanks
10Gen introduce Mongodb Connector. You can integrate Mongodb with Solr using this tool.
Blog post : Introducing Mongo Connector
Github page : mongo-connector
I have created a plugin to allow you to load data from MongoDb using the Solr data import handler.
Check it out at:
https://github.com/james75/SolrMongoImporter
I wrote a response to a similar question, except it was how to import data from MySQL into SOLR. The example code is in PHP, but should give you a general idea. All you would need to do is set up an iterator to step through your MongoDB assets, extract the data to SOLR datatypes, and then save it to your SOLR index.
If you want it to be real-time, you could add some custom code to the save mechanism (assuming this can be done with MongoDB), and save directly to the SOLR index, then run a commit script to commit data every 15 minutes (via cron).