logstash mongodb output and ISODate type - mongodb

I have some troubles trying to convert a date type field into mongoDB format (ISODate).
I have a RabbitMQ queue with JSON messages inside. These messages have a Date property like this :
Date : "2014-05-01T14:53:34.25677Z"
My logstash service read the RabbitMQ queue and inject messages into mongoDB.
Here is my logstash config file :
input {
rabbitmq {
...
codec => json
}
}
output {
mongodb {
codec => json
collection => "log"
isodate => true
database => "Test"
uri => "mongodb://localhost:27017"
}
}
My problem is that my Date property is insterted as string instead as Date. How can I do to tell Logstash to insert my Date field as an ISODate field into mongoDB?
Thank you

You should use a logstash Date filter to convert the string into a Date prior to inserting it into MongoDB: http://logstash.net/docs/1.4.2/filters/date
Don't know your full schema but it should looking something like this:
filter {
date {
match => [ "Date", "ISO8601" ]
}
}
Note the use of "ISO8601" - that appears to match the format you are receiving but you may need to play around a bit with it. As you test this I'd strongly suggest using the stdout output option for test runs to easily see what's getting done prior to insertion into MongoDB:
output {
stdout { codec => rubydebug }
}

Related

sync data from mongodb to elasticsearch via logstash

I just want to sync the data from mongodb to elastic using logstash. Its working good, when any new record comes in mongodb, logstash pushes into elastic. But when I update any record in mongodb then it does not change into elasticsearch even when i delete it nothing happen . I want to make changes in the config file so that when any record updates or deleted in mongo it should reflect in elastic as well.
input {
mongodb {
uri => 'mongodb://xxxxxx:32769/database'
placeholder_db_dir =>'/usr/share/logstash/bin/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'tags'
}
}
filter {
mutate {
rename => { "_id" => "mongo_id" }
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_data"
hosts => ["https://xxxxxxxx:8443"]
ssl => true
doc_as_upsert => true
}
}
That is not a use-case that the input was designed to support. The documentation states "This was designed for parsing logs that were written into mongodb. This means that it may not re-parse db entries that were changed and already parsed." For "may not" read "will not". The code builds a cursor that finds documents with an id greater than the last id it read. It never looks for updates or deletions. Note also that the test is "greater than the last id" and the way it initializes the last id means it never reads the first document in the collection.

Syncing the data from mongoDB to elasticsearch using Logstash

I am trying to sync the mongoDB database to the elasticsearch. I am using logstash-input-mongoDb and logstash-output-elasticsearch plugins.
The issue is mongoDb plugin is not able to extract all the information from the inserted document in mongodb, thus I am seeing only few fields being inserted to the elasticsearch. And I also get the entire query as the log in elasticsearch index. I tried to manipulate the filters in the config file for the logstash and change the input to the elasticsearch but could not make it work.
Any help or suggestion would be great.
Edit:
Mongo schema:
A:{
B: 'sometext',
C: {G: 'someText', H:'some text'}
},
D:[
{E:'sometext',F:'sometext'},
{E:'sometext',F:'sometext'},
{E:'sometext',F:'sometext'}
]
plugin:
input {
mongodb {
uri => 'mongodb://localhost:27017/testDB'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'testCOllection'
batch_size => 1000
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "testdb_testColl"
hosts => ["localhost:9200"]
}
}
output to elastic:
{
//some metadata
A_B: 'sometext',
A_C_G: 'someText',
A_C_H: 'some text',
log_entry: 'contains complete document inserted to mongoDB'
}
We are not getting property D of mongo collection in the elastic.
Hope this explains the problem more elaborately.
because your configuration looked good to me, I checked the issues of the phutchins/logstash-input-mongodb repo, and I found this one: "array not stored to elasticsearch", which pretty much described your problem. It is still an open issue, but you might want to try out the workaround suggested by ivancruzbht. Such workaround uses the ruby Logstash filter to parse the log_entry field, which you also confirmed has all the fields - including D.

Parse Server aggregate using match on date column

I'm using parse-server version 2.7.4 with parse (the JavaScript interface) version 1.11.1. I'm trying to execute an aggregate query using match on a date column. Here's what my pipeline looks like:
[
{
match: {
'_created_at': new Date('2018-04-01T00:00:00.000Z')
}
}
]
I most definitely have records in the database greater than that date, yet this query returns me zero records in JavaScript. However, if I run it in the Mongo shell, I get documents returned. Here is what I execute in the Mongo shell:
[
{
$match: {
'_created_at': ISODate('2018-04-01T00:00:00.000Z')
}
}
]
Am I doing something wrong with the parse JavaScript API or does it not support match aggregation on date columns? Thanks for any help.

How can I specify the fileds to insert in MongoDB Output in Logstash

I've below in output config, but it inserts the raw data in MongoDB. How can I insert only a few selected fields? For InfluxDB, we have an attribute data_points with which I can do it, but MongoDB plugin doesn't seem to have any such feature.
mongodb {
collection => "logs"
database => "test"
uri => "mongodb://localhost:27017"
codec => line
{
enable_metric => "false"
format => "data1:%{val1}, data2:%{val2}, data3:%{val3}"
}
}
You can use the Logstash prune filter for this. The prune filter's whitelist_names setting allows you to removes all fields that are not enumerated in the array.
filter {
prune {
whitelist_names => ["field1", "field2", "field3"]
}
}
Something that I think is really cool about the prune filter is that it also allows you to input regular expressions - and removes any field that does not match the regular expression. So instead of the above, you could have:
filter {
prune {
whitelist_names => ["^field\d+"]
}
}
Another note: The prune filter does not come installed by default. You must run bin/logstash-plugin install logstash-filter-prune

How to Convert mongodb ISODate to string in mongoDB?

I have my ISODate in mongo as ISODate and I want to just that in string format with a specific datetime format.
Here is the ISODate:
ISODate("2020-04-24T11:41:47.280Z")
Expected Result:
"2020-04-24T11:41:47.280Z"
I want this to be happened on mongodb only as many of my services are expecting in this format, and I don't want to make changes in all services as its a tedious job.
I got the expected result while i was trying the following.
ISODate("2020-04-24T11:41:47.280Z").toJSON()
This will give me back the string
"2020-04-24T11:41:47.280Z"
perhaps simply convert a date into string? this has less to do with mongo than js. moment is awesome but unusable in a mongo shell script.
db.events.find().forEach(function(doc) {
// printjson ("Document is " + doc);
var isoDate = doc.t; // t is correct key?
var isoString = isoDate.toISOString()
// update the collection with string using a new key
db.events.update(
{"_id":doc._id},
{
$set: {"iso_str":isoString}
}
);
// or overwrite using 't' key db.events.update({"_id":doc._id},{$set:{"t":isoString}});
})
I just came across this issue. There isn't anything built into ISODate turns out. So I'm converting the ISODate to JSON text, and then I do a substring to get the part I want. You can also use method on ISODate to get year, month, date separately and then combine them.
function formatDate(isoDate){
return isoDate.toJSON().substr(9, 20);
}
I assume that what you need is the dateToString function
https://docs.mongodb.com/manual/reference/operator/aggregation/dateToString/
If you have mixed data types[string & ISODate()] in an attribute & still want to normalize them to one data type e.g., 'string' then
typeof attr_val === 'string' ? attr_val : attr_val.toJSON()
References:
#manu answer
Typechecking
typeof
db.inventries.find().forEach(function (doc) {
let mydate = new Date(doc.date).toISOString();
db.inventries.update({ "_id": doc._id }, { $set: { "date": ISODate(mydate) } })
})