I just want to sync the data from mongodb to elastic using logstash. Its working good, when any new record comes in mongodb, logstash pushes into elastic. But when I update any record in mongodb then it does not change into elasticsearch even when i delete it nothing happen . I want to make changes in the config file so that when any record updates or deleted in mongo it should reflect in elastic as well.
input {
mongodb {
uri => 'mongodb://xxxxxx:32769/database'
placeholder_db_dir =>'/usr/share/logstash/bin/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'tags'
}
}
filter {
mutate {
rename => { "_id" => "mongo_id" }
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_data"
hosts => ["https://xxxxxxxx:8443"]
ssl => true
doc_as_upsert => true
}
}
That is not a use-case that the input was designed to support. The documentation states "This was designed for parsing logs that were written into mongodb. This means that it may not re-parse db entries that were changed and already parsed." For "may not" read "will not". The code builds a cursor that finds documents with an id greater than the last id it read. It never looks for updates or deletions. Note also that the test is "greater than the last id" and the way it initializes the last id means it never reads the first document in the collection.
Related
I am trying to sync the mongoDB database to the elasticsearch. I am using logstash-input-mongoDb and logstash-output-elasticsearch plugins.
The issue is mongoDb plugin is not able to extract all the information from the inserted document in mongodb, thus I am seeing only few fields being inserted to the elasticsearch. And I also get the entire query as the log in elasticsearch index. I tried to manipulate the filters in the config file for the logstash and change the input to the elasticsearch but could not make it work.
Any help or suggestion would be great.
Edit:
Mongo schema:
A:{
B: 'sometext',
C: {G: 'someText', H:'some text'}
},
D:[
{E:'sometext',F:'sometext'},
{E:'sometext',F:'sometext'},
{E:'sometext',F:'sometext'}
]
plugin:
input {
mongodb {
uri => 'mongodb://localhost:27017/testDB'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'testCOllection'
batch_size => 1000
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "testdb_testColl"
hosts => ["localhost:9200"]
}
}
output to elastic:
{
//some metadata
A_B: 'sometext',
A_C_G: 'someText',
A_C_H: 'some text',
log_entry: 'contains complete document inserted to mongoDB'
}
We are not getting property D of mongo collection in the elastic.
Hope this explains the problem more elaborately.
because your configuration looked good to me, I checked the issues of the phutchins/logstash-input-mongodb repo, and I found this one: "array not stored to elasticsearch", which pretty much described your problem. It is still an open issue, but you might want to try out the workaround suggested by ivancruzbht. Such workaround uses the ruby Logstash filter to parse the log_entry field, which you also confirmed has all the fields - including D.
I have a mongodb collection called orders and i am running the following query on it
module.exports.readPendingOrders = function(count){
return new Promise((resolve, reject) => {
Order.find({status: "pending"})
.sort({created: -1})
.limit(100)
.skip(count)
.exec()
.then(rem => {
resolve(rem);
})
.catch(error => {
reject(error.message)
})
})
}
I have this same paginated format trying to get all documents and to get all documents with status : "completed".
The problem is that it always returns one less document than i am expecting.
It always skips the last record in the database, and i have about 8 records in the database.
The front end is a react app and i didnt notice it there till i was testing the query in Postman and i saw that it returned one less document than i expected.
Not sure what is wrong here.
I've below in output config, but it inserts the raw data in MongoDB. How can I insert only a few selected fields? For InfluxDB, we have an attribute data_points with which I can do it, but MongoDB plugin doesn't seem to have any such feature.
mongodb {
collection => "logs"
database => "test"
uri => "mongodb://localhost:27017"
codec => line
{
enable_metric => "false"
format => "data1:%{val1}, data2:%{val2}, data3:%{val3}"
}
}
You can use the Logstash prune filter for this. The prune filter's whitelist_names setting allows you to removes all fields that are not enumerated in the array.
filter {
prune {
whitelist_names => ["field1", "field2", "field3"]
}
}
Something that I think is really cool about the prune filter is that it also allows you to input regular expressions - and removes any field that does not match the regular expression. So instead of the above, you could have:
filter {
prune {
whitelist_names => ["^field\d+"]
}
}
Another note: The prune filter does not come installed by default. You must run bin/logstash-plugin install logstash-filter-prune
I have some troubles trying to convert a date type field into mongoDB format (ISODate).
I have a RabbitMQ queue with JSON messages inside. These messages have a Date property like this :
Date : "2014-05-01T14:53:34.25677Z"
My logstash service read the RabbitMQ queue and inject messages into mongoDB.
Here is my logstash config file :
input {
rabbitmq {
...
codec => json
}
}
output {
mongodb {
codec => json
collection => "log"
isodate => true
database => "Test"
uri => "mongodb://localhost:27017"
}
}
My problem is that my Date property is insterted as string instead as Date. How can I do to tell Logstash to insert my Date field as an ISODate field into mongoDB?
Thank you
You should use a logstash Date filter to convert the string into a Date prior to inserting it into MongoDB: http://logstash.net/docs/1.4.2/filters/date
Don't know your full schema but it should looking something like this:
filter {
date {
match => [ "Date", "ISO8601" ]
}
}
Note the use of "ISO8601" - that appears to match the format you are receiving but you may need to play around a bit with it. As you test this I'd strongly suggest using the stdout output option for test runs to easily see what's getting done prior to insertion into MongoDB:
output {
stdout { codec => rubydebug }
}
Does MongoDB support multiple document update if selector (first argument) is selecting more than 1 document inside a collection.
In the below example first one works fine as it selects only a particular document and modifies zip value.
While in second case collection $addresses has multiple documents which has 'home' => 'canada', it doesn't update anything.
Can someone please help me ?
$addresses->update(array('_id' => new MongoId('4f69de380c211d6c21000001')),
array('$set' => array('zip' => 20)));
$addresses->update(array('home' => 'canada')),
array('$set' => array('zip' => 20)));
Edit:
Equivalent javascript command
db.addresses.update({home: "canada"}, {$set: {zip: 20}})
Updates zip value of first encountered match, is this the expected behavior.
console command is updating at least one document, PHP is not doing anything if selector matches more than 1 document.
As long as the second statement matches anything it should always update a single document.
If you want it to apply to multiple documents you should pass in the multiple flag as documented here.
http://www.php.net/manual/en/mongocollection.update.php
In your case the query should read:
$addresses->update(array('home' => 'canada'),
array('$set' => array('zip' => 20)),
array('multiple' => true)
);