Elasticsearch and MongoDB: no river _meta document found after 5 attempts - mongodb

I have a MongoDB database named news to which I tried to index with ES.
Using these plugins:
richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9
and elasticsearch/elasticsearch-mapper-attachments/2.5.0
This is what happening when I tried to create the index. I have tried to delete the index and recreating it, without that helping.
$ curl -XPUT 'http://localhost:9200/_river/news/_meta' -d #init.json
init.json
{
"type": "mongodb",
"mongodb": {
"db": "news",
"collection": "entries"
},
"index": {
"name": "news",
"type": "entries"
}
}
Here is a log
update_mapping [mongodb] (dynamic)
MongoDB River Plugin - version[2.0.9] - hash[73ddea5] - time[2015-04-06T21:16:46Z]
setRiverStatus called with mongodb - RUNNING
river mongodb startup pending
Starting river mongodb
MongoDB options: secondaryreadpreference [false], drop_collection [false],
include_collection [], throttlesize [5000], gridfs [false], filter [null],
db [news], collection [entries], script [null], indexing to [news]/[entries]
MongoDB version - 3.0.2
update_mapping [mongodb] (dynamic)
[org.elasticsearch.river.mongodb.CollectionSlurper] Cannot ..
import collection entries into existing index
d with mongodb - INITIAL_IMPORT_FAILED
Started river mongodb
no river _meta document found after 5 attempts
no river _meta document found after 5 attempts
Any suggestions to what might be wrong?
I'm running ES 1.5.2 and MongoDB 3.0.2 on OS X.

On the mongodb river github pages, it looks like the plugin is supported up until version 1.4.2, but not higher (i.e. you're running 1.5.2)
Also note that rivers have been deprecated in ES v1.5 and there's an open issue in the mongodb river project on this very topic.
UPDATE after chatting with #martins
Finally, the issue was simply that the name of the created river was wrong (i.e. news instead of mongodb), the following command would properly create the mongodb river, which still works with ES 1.5.2 even though not it's officially tested.
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d #init.json

Related

Type error on mongodb 3

When i try to insert data on mongodb 3 through command line it's showing following error
use video;
switched to db video
db.movies.insertOne({ "title": "Jaws", "year": 1975, "imdb": "tt0073195" });
2018-03-26T12:42:42.233+0530 E QUERY TypeError: Property 'insertOne' of object video.movies is not a function at (shell):1:11`
but video db also not created
Please help me to rectify this problem.
MongoDB supports db.collection.insertOne() from version 3.2, please check your mongodb version by using the mongo shell command
db.version()
References:
insertOne
version
Try with db.movies.insert instead of db.movies.insertOne and check If it's working fine. If it's working then your mongo version is less than 3.2. If not then share your mongoDb console Screenshot.

How to create an index with ElasticSearch, River and MongoDB?

I used this tutorial to install and configure MongoDB / Elasticsearch.
The whole tutorial worked on Mac OSX Yosemite and now I tried to do the same on Ubuntu 14.04.
Here is my ElasticSearch log:
[2014-12-08 15:49:13,733][INFO ][cluster.service ] [Western Kid] new_master [Western Kid][fo8GLpDoRyKYBAkjk7f-jw][my_hostname][inet[localhost/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2014-12-08 15:49:13,758][INFO ][http ] [Western Kid] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2014-12-08 15:49:13,758][INFO ][node ] [Western Kid] started
[2014-12-08 15:49:14,449][INFO ][gateway ] [Western Kid] recovered [1] indices into cluster_state
[2014-12-08 15:49:15,225][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb
[2014-12-08 15:49:15,230][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.4] - hash[7472875] - time[2014-11-11T13:26:19Z]
[2014-12-08 15:49:15,231][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [my_db_name], collection [my_collection], script [null], indexing to [my_index]/[my_type]
[2014-12-08 15:49:15,231][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - RUNNING
[2014-12-08 15:57:56,543][INFO ][cluster.metadata ] [Western Kid] [_river] update_mapping [my_db_name] (dynamic)
When I tried to start indexing my collection, I got the following message:
{
"_index": "_river",
"_type": "my_type",
"_id": "_meta",
"_version": 4,
"created": false
}
The version is increasing every time I try and started with 1.
I guess, created:false means, the index could not be created for some reason but I have no idea why.
The version of the river is increasing because you are actually updating the _river index so Elasticsearch is not creating it since it's already there.
You might want to install elasticsearch-head plugin to visualize your cluster better since you don't seem to be very familiar with the API.
Try to delete the _river and create it again and you'll see that this time it will actually have the status created.

elasticsearch throw exception work with mongodb river

I followed the link http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-and-Mongo-DB-td4033358.html to integrate elasticsearch and mongodb using mongodb river. The versions of each component are:
ubuntu 12.04 64bit
ES 0.90.0
mongodb 2.4.3
river 1.6.5
Mongodb is standalone running in one server but according to this link http://loosexaml.wordpress.com/2012/09/03/how-to-get-a-mongodb-oplog-without-a-full-replica-set/, the oplog is opened as replSet and oplogSize is configured in /etc/mongodb.conf, and the db.oplog.rs.find() also displayed some operation records.
The index added by:
curl -XPUT localhost:9200/_river/appdata/_meta -d'
{
"type": "mongodb",
"mongodb" : {
"db": "test_appdata",
"collection": "app_collection"
},
"index": {
"name": "test_appdata",
"type": "app"
}
}'
But when the elasticsearch started, the log show some exception as follow:
[2013-05-07 23:20:40,400][INFO ][river.mongodb ] [Ransak the Reject] [mongodb][app] starting mongodb stream. options: secondaryreadpreference [false], throttlesize [500], gridfs [false], filter [], db [test_appdata], script [null], indexing to [test_appdata]/[app]
Exception in thread "elasticsearch[Sundragon][mongodb_river_slurper][T#1]" java.lang.NoSuchMethodError: org.elasticsearch.action.get.GetResponse.exists()Z
at org.elasticsearch.river.mongodb.MongoDBRiver.getLastTimestamp(MongoDBRiver.java:1088)
at org.elasticsearch.river.mongodb.MongoDBRiver.access$2200(MongoDBRiver.java:93)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.getIndexFilter(MongoDBRiver.java:967)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.oplogCursor(MongoDBRiver.java:1021)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:858)
at java.lang.Thread.run(Thread.java:679)
I'm a newbie to elasticsearch and mongodb, is the replica setting of mongodb caused the error?
Any suggestion is appreciated.
Your river is not compatible with Elasticsearch 0.90.
Move to ES 0.20.6 or ask for a patch in Mongodb river Project.

Updating mongodb references errors out due to field names

I have a MongoDB collection and I'm trying to update all the entries in it to change the name of a field used to store a reference. The Query I'm using is
db.products.find().forEeach(function(p) {
p.newField = p.oldField;
db.products.save(p);
});
The problem is that p.oldField is a DBRef following the standard format of { "$ref": "collection", "$id": ObjectId("...")}. When I try to run the db.products.save(p); Mongo returns the following error:
Sat Oct 1 13:00:57 uncaught exception: field names cannot start with $ [$db]
I'm using version 1.8.2 of the MongoDB shell. I have seen this work on an older version of the shell (1.6.5), which is where I originally came up with this query. But I can't seem to make this work on newer versions.

mongodb river for elasticsearch

Is there any official mongodb river available for elasticsearch ? I am using mongodb in node.js through the module mogoose.
I have seen one in http://www.matt-reid.co.uk/blog_post.php?id=68
Is this the correct one ? It says unofficial though...
Edit:
looks like, https://github.com/aparo/elasticsearch has inbuilt mongodb plugin.. Is there any doc available about how to configure this with mongodb and how mongodb pushes data for indexing to elasticsearch?
There is a new MongoDB river on github:
https://github.com/richardwilly98/elasticsearch-river-mongodb
according to the code you can specify several things but there is no separate doc (expect one mailing list discussion):
https://github.com/aparo/elasticsearch/blob/master/plugins/river/mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java
https://github.com/aparo/elasticsearch/blob/master/plugins/river/mongodb/src/test/java/org/elasticsearch/river/mongodb/MongoDBRiverTest.java
This isn't really the answer you're looking for. I looked at building this mongo river but I found some discussion on it having some memory leaks and I didn't want to fiddle with Java code. I wrote my own mongo->ES importer using the bulk API.
It's a work in progress, so feel free to contribute! :)
https://github.com/orenmazor/elastic-search-loves-mongo
Yes, There is a new MongoDB river on github:
https://github.com/richardwilly98/elasticsearch-river-mongodb
For Further Explanation You can follow below steps:
Step.1: -Install
ES_HOME/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.4.0
ES_HOME/bin/plugin -install richardwilly98/elasticsearch-river-mongodb/1.4.0
Step.2: -Restart Elasticsearch
ES_HOME/bin/service/elasticsearch restart
Step.3: -Enable replica sets in mongodb
go to mongod.conf & Add line
replSet=rs0
save & Exit
Restart mongod
Step.4: -Tell elasticsearch to index the “person” collection in testmongo database by issuing the following command in your terminal
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'
Step.5: -add some data to the mongodb through mongo terminal
use testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)
Step.6: -Use this command to search the data
curl -XGET 'http://localhost:9200/mongoindex/_search?q=firstName:John'
NOTE:
DELETE /_river
DELETE/_mongoindex
Again run this command,
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'
Step.7: -See HQ Plugin
In mongoindex, you will get your data.