I followed the link http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-and-Mongo-DB-td4033358.html to integrate elasticsearch and mongodb using mongodb river. The versions of each component are:
ubuntu 12.04 64bit
ES 0.90.0
mongodb 2.4.3
river 1.6.5
Mongodb is standalone running in one server but according to this link http://loosexaml.wordpress.com/2012/09/03/how-to-get-a-mongodb-oplog-without-a-full-replica-set/, the oplog is opened as replSet and oplogSize is configured in /etc/mongodb.conf, and the db.oplog.rs.find() also displayed some operation records.
The index added by:
curl -XPUT localhost:9200/_river/appdata/_meta -d'
{
"type": "mongodb",
"mongodb" : {
"db": "test_appdata",
"collection": "app_collection"
},
"index": {
"name": "test_appdata",
"type": "app"
}
}'
But when the elasticsearch started, the log show some exception as follow:
[2013-05-07 23:20:40,400][INFO ][river.mongodb ] [Ransak the Reject] [mongodb][app] starting mongodb stream. options: secondaryreadpreference [false], throttlesize [500], gridfs [false], filter [], db [test_appdata], script [null], indexing to [test_appdata]/[app]
Exception in thread "elasticsearch[Sundragon][mongodb_river_slurper][T#1]" java.lang.NoSuchMethodError: org.elasticsearch.action.get.GetResponse.exists()Z
at org.elasticsearch.river.mongodb.MongoDBRiver.getLastTimestamp(MongoDBRiver.java:1088)
at org.elasticsearch.river.mongodb.MongoDBRiver.access$2200(MongoDBRiver.java:93)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.getIndexFilter(MongoDBRiver.java:967)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.oplogCursor(MongoDBRiver.java:1021)
at org.elasticsearch.river.mongodb.MongoDBRiver$Slurper.run(MongoDBRiver.java:858)
at java.lang.Thread.run(Thread.java:679)
I'm a newbie to elasticsearch and mongodb, is the replica setting of mongodb caused the error?
Any suggestion is appreciated.
Your river is not compatible with Elasticsearch 0.90.
Move to ES 0.20.6 or ask for a patch in Mongodb river Project.
Related
I am trying to sync my Postgres database with Aws Elasticsearch using PGSync
I have defined a simple schema:
[
{
"database": "tenancyportal",
"index": "properties",
"nodes": [
{
"table": "properties",
"schema": "public",
"columns": ["id", "address"]
}
]
}
]
But when I am trying to bootstrap the database using
bootstrap --config schema.json
I get the following error:
elasticsearch.exceptions.NotFoundError: NotFoundError(404,
'index_not_found_exception', 'no such index [:9200]', :9200,
index_or_alias)
In the below screenshot, you will be able to see the GET URL for elasticsearch is completely wrong, I am not able to understand what config is causing it to be formed like that.
It looks like your AWS Elasticsearch URL is not constructed properly. This was adressed in a recent update to PGSync. Can you pull the latest master branch and try this again.
I use the Microsoft Azure Comos Emulator version 2.11.11.0. I start it with the command line:
CosmosDB.Emulator.exe /EnableMongoDbEndpoint=3.6
If I connect with my application to the MongoDB endpoint I receive the follow error:
com.mongodb.MongoCommandException: Command failed with error 1 (InternalError):
'[ActivityId=64bdb726-f138-4025-8d5b-ca23bbda384c] Internal error.' on server localhost:10255.
The full response is {"ok": 0.0, "errmsg": "[ActivityId=64bdb726-f138-4025-8d5b-ca23
bbda384c] Internal error.", "code": 1, "codeName": "InternalError"}
What this error means? What I need to do else to use the MongoDB API?
Failing ist the first write command:
'{"createIndexes": "fs.files", "indexes": [{"key": {"filename": 1}, "name": "filename_1", "ns": "admin.fs.files"}], "$db": "admin", "lsid": {"id": {"$binary": {"base64": "lFHKYPJ0Rq+ip587pl8YPQ==", "subType": "04"}}}, "$readPreference": {"mode
": "primaryPreferred"}}'
It was produces from the simple Java line:
collection.createIndex( new BasicDBObject( FILENAME, 1 ) );
PS: With MongoDb endpoint 3.2 this is working. It must be any problem with the emulator.
With the emulator Mongo API 3.6 there must be created the collection explicit. With the original MongoDB and the Mongo API end point 3.2 the collection will be indirect created if a Index is created or data are saved.
I have a MongoDB database named news to which I tried to index with ES.
Using these plugins:
richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9
and elasticsearch/elasticsearch-mapper-attachments/2.5.0
This is what happening when I tried to create the index. I have tried to delete the index and recreating it, without that helping.
$ curl -XPUT 'http://localhost:9200/_river/news/_meta' -d #init.json
init.json
{
"type": "mongodb",
"mongodb": {
"db": "news",
"collection": "entries"
},
"index": {
"name": "news",
"type": "entries"
}
}
Here is a log
update_mapping [mongodb] (dynamic)
MongoDB River Plugin - version[2.0.9] - hash[73ddea5] - time[2015-04-06T21:16:46Z]
setRiverStatus called with mongodb - RUNNING
river mongodb startup pending
Starting river mongodb
MongoDB options: secondaryreadpreference [false], drop_collection [false],
include_collection [], throttlesize [5000], gridfs [false], filter [null],
db [news], collection [entries], script [null], indexing to [news]/[entries]
MongoDB version - 3.0.2
update_mapping [mongodb] (dynamic)
[org.elasticsearch.river.mongodb.CollectionSlurper] Cannot ..
import collection entries into existing index
d with mongodb - INITIAL_IMPORT_FAILED
Started river mongodb
no river _meta document found after 5 attempts
no river _meta document found after 5 attempts
Any suggestions to what might be wrong?
I'm running ES 1.5.2 and MongoDB 3.0.2 on OS X.
On the mongodb river github pages, it looks like the plugin is supported up until version 1.4.2, but not higher (i.e. you're running 1.5.2)
Also note that rivers have been deprecated in ES v1.5 and there's an open issue in the mongodb river project on this very topic.
UPDATE after chatting with #martins
Finally, the issue was simply that the name of the created river was wrong (i.e. news instead of mongodb), the following command would properly create the mongodb river, which still works with ES 1.5.2 even though not it's officially tested.
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d #init.json
I used this tutorial to install and configure MongoDB / Elasticsearch.
The whole tutorial worked on Mac OSX Yosemite and now I tried to do the same on Ubuntu 14.04.
Here is my ElasticSearch log:
[2014-12-08 15:49:13,733][INFO ][cluster.service ] [Western Kid] new_master [Western Kid][fo8GLpDoRyKYBAkjk7f-jw][my_hostname][inet[localhost/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2014-12-08 15:49:13,758][INFO ][http ] [Western Kid] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2014-12-08 15:49:13,758][INFO ][node ] [Western Kid] started
[2014-12-08 15:49:14,449][INFO ][gateway ] [Western Kid] recovered [1] indices into cluster_state
[2014-12-08 15:49:15,225][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb
[2014-12-08 15:49:15,230][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.4] - hash[7472875] - time[2014-11-11T13:26:19Z]
[2014-12-08 15:49:15,231][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [my_db_name], collection [my_collection], script [null], indexing to [my_index]/[my_type]
[2014-12-08 15:49:15,231][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - RUNNING
[2014-12-08 15:57:56,543][INFO ][cluster.metadata ] [Western Kid] [_river] update_mapping [my_db_name] (dynamic)
When I tried to start indexing my collection, I got the following message:
{
"_index": "_river",
"_type": "my_type",
"_id": "_meta",
"_version": 4,
"created": false
}
The version is increasing every time I try and started with 1.
I guess, created:false means, the index could not be created for some reason but I have no idea why.
The version of the river is increasing because you are actually updating the _river index so Elasticsearch is not creating it since it's already there.
You might want to install elasticsearch-head plugin to visualize your cluster better since you don't seem to be very familiar with the API.
Try to delete the _river and create it again and you'll see that this time it will actually have the status created.
Is there any official mongodb river available for elasticsearch ? I am using mongodb in node.js through the module mogoose.
I have seen one in http://www.matt-reid.co.uk/blog_post.php?id=68
Is this the correct one ? It says unofficial though...
Edit:
looks like, https://github.com/aparo/elasticsearch has inbuilt mongodb plugin.. Is there any doc available about how to configure this with mongodb and how mongodb pushes data for indexing to elasticsearch?
There is a new MongoDB river on github:
https://github.com/richardwilly98/elasticsearch-river-mongodb
according to the code you can specify several things but there is no separate doc (expect one mailing list discussion):
https://github.com/aparo/elasticsearch/blob/master/plugins/river/mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java
https://github.com/aparo/elasticsearch/blob/master/plugins/river/mongodb/src/test/java/org/elasticsearch/river/mongodb/MongoDBRiverTest.java
This isn't really the answer you're looking for. I looked at building this mongo river but I found some discussion on it having some memory leaks and I didn't want to fiddle with Java code. I wrote my own mongo->ES importer using the bulk API.
It's a work in progress, so feel free to contribute! :)
https://github.com/orenmazor/elastic-search-loves-mongo
Yes, There is a new MongoDB river on github:
https://github.com/richardwilly98/elasticsearch-river-mongodb
For Further Explanation You can follow below steps:
Step.1: -Install
ES_HOME/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.4.0
ES_HOME/bin/plugin -install richardwilly98/elasticsearch-river-mongodb/1.4.0
Step.2: -Restart Elasticsearch
ES_HOME/bin/service/elasticsearch restart
Step.3: -Enable replica sets in mongodb
go to mongod.conf & Add line
replSet=rs0
save & Exit
Restart mongod
Step.4: -Tell elasticsearch to index the “person” collection in testmongo database by issuing the following command in your terminal
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'
Step.5: -add some data to the mongodb through mongo terminal
use testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)
Step.6: -Use this command to search the data
curl -XGET 'http://localhost:9200/mongoindex/_search?q=firstName:John'
NOTE:
DELETE /_river
DELETE/_mongoindex
Again run this command,
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'
Step.7: -See HQ Plugin
In mongoindex, you will get your data.