I'm storing date as string in "mm/dd/yyyy" format. I want to sort by this date field. I tried below query with few test data.
db.collection.find().sort({date: -1}).pretty()
and it is working fine. Would this work fine ever or should i convert this to MongoDate for reliable sorting?
Within MongoDB 3.6 you can now use $dateFromString (https://docs.mongodb.com/manual/reference/operator/aggregation/dateFromString/)
db.logmessages.aggregate( [ {
$project: {
date: {
$dateFromString: {
dateString: '$date'
}
}
}
}, { $sort: { date : 1} } ] )
With yyyy being last, that sort isn't going to work across years.
Probably best to switch to a yyyy-mm-dd formatted string or an actual Date type. Both of those will sort correctly.
Related
Here, In my mongo collection, the date type column data is stored as string. I have a view where I need to compare this date with current date and time $$NOW. Since my date is stored as string. The query is getting executed but not getting compared. I tried to convert this string to date type but the data changed like this 2022-05-23 1:28:36 quotes were gone. Tried to run same query, but getting an error.
sample data:
[{oldDate:"2022-05-23 1:28:36"}]
My query:
db.collection.aggregate([
{
"$match": {
$expr: {
$lt: [
"$oldDate",
"$$NOW"
]
}
}
}
])
If the data in db is in the form of ISODate("2022-05-23 1:28:36") then this query is working. But the actual data in my db for this column is in the form of string. Can anyone help me to convert this to ISODate() through code itself and make comparison work.
Use $toDate operator to convert date string to date.
db.collection.aggregate([
{
"$match": {
$expr: {
$lt: [
{
$toDate: "$oldDate"
},
"$$NOW"
]
}
}
}
])
Sample Mongo Playground
I have some logs that read by Filebeat, filter in Logstash and sent to MongoDB. Filebeat generate #timestamp field on logs, the log might look like this
{
"_id" : ObjectId("622884e8ed814f1590000076"),
"#timestamp" : "\"2022-03-09T10:43:46.000\"",
"stream" : "stderr",
"message" : "[2022-03-09T10:43:46.528612+00:00] testing.INFO: Message error [] []"
}
but the #timestamp field write as String not Date. I have to read the timestamp as ISODate on MongoDB. It might look like this
"#timestamp": ISODate("2022-03-10T01:43:46.000Z")
instead of
"#timestamp" : "\"2022-03-09T10:43:46.000\""
Any suggestion how to change the datatype from string into date?
UPDATE
Trying to using match on date filter.
date {
match => [ "#timestamp", "MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss", "ISO8601" ]
}
}
on mongo the datatype still not change. I also tried to make another field with current date (logstash_processed_at) with this
ruby {
code => "event.set('logstash_processed_at', Time.now());"
}
And using the date match too, but on the mongo the datatype still string
Here is how to convert the timestamps already in mongoDB:
db.collection.update({},
[
{
$addFields: {
"#timestamp": {
$dateFromString: {
dateString: {
$substr: [
"$#timestamp",
1,
23
]
}
}
}
}
}
],
{
multi: true
})
playground
But you need to fix the injestion and use the logstash filter in order to convert the date in string to date in ISODate():
filter {
date {
match => [ "logdate", "MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss", "ISO8601" ]
}
}
Also check here the monggodb output plugin:
isodate
Value type is boolean
Default value is false
If true, store the #timestamp field in MongoDB as an ISODate type instead of an ISO8601 string.
Instead of changing the filter it solved by adding new field on output mongodb at logstash config. the field is isodate => true.
output {
mongodb {
...
isodate => true
}
}
This field change the #timestamp field from string to ISODate() format when it sent to mongodb. If you are looking for how changing the timestamp into ISODate() in mongodb, #R2D2 has the answer up there.
i've to run a query like this (sql) in MongoDb 4:
SELECT * FROM log WHERE DATE_ADD(created_at, INTERVAL 2 HOUR) < NOW()
Basically, I want to find all the documents, in the PENDING state, whose creation date PLUS TWO HOURS is less than now .. Let me explain: I want to find all the documents in the PENDING state that have been in PENDING for more than two hours.
I feel stupid, but I am failing to do this with MongoDb.
I also created a playground:
https://mongoplayground.net/p/4bifqiX2KMJ
Can you help me?
You can add hours in ISO date using $add, convert string date to ISO date using dateFromString,
let date = new Date();
db.collection.find({
status: "pending",
$expr: {
$lt: [
{
$add: [
// convert string date to ISOdate, if its already then use only "$inserted_at"
{ $dateFromString: { dateString: "$inserted_at" } },
// add milliseconds
7200000 // (60*60*2000)
]
},
date
]
}
})
Playground
Or subtract from current date and then compare the condition,
let date = new Date();
date = new Date(date.getHours()-2); //subtract 2 hours
db.collection.find({
status: "pending",
$expr: {
$lt: [
{ $dateFromString: { dateString: "$inserted_at" } },
date
]
}
})
Playground
So currently store start and end time that a user listens to our stations in epoch format, however now I need to do a query that groups listeners if they where listening with an hour block.
For example:
last 7 days Query would need to go back 7 days normally I would just do something like this
items.find({
history.$.starttime: {
$gte: ISODate("2020-07-10T00:00:00.000Z"),
$lt: ISODate("2020-009-17T24:59:59.000Z")
}})
However we don't store the date in this format we store it looking like this.
So how do a search between dates if the dates are in epoch format?
I have tried
{'history': {'$elemMatch':{'starttime': {'$gte':1592966066060}}}
UPDATE:
this works
{"history.starttime":{$gte:1593236606706}}
You can convert timestamp to ISO date using $toDate, like this,
db.collection.aggregate([
{
$addFields: {
startTimeDate: {
$toDate: "$startTime"
},
endTimeDate: {
$toDate: "$endTime"
}
}
},
{
$match: {
startTimeDate: {
$gte: ISODate("2020-01-20T21:20:00Z")
}
}
}
])
You can removed that added field using:
{
$project: {
startTimeDate: 0,
endTimeDate: 0
}
}
Working Playground: https://mongoplayground.net/p/XiEYVxvmfpv
Convert the times to Unix timestamps in your application, specify the timestamps in the query.
I have data inserted in UTC time format in mongodb. I want timings to be converted based on the timezone. Is there any possibility to do so in mongo query?
In mongo version 3.6 timezone has been added, mongo doc
expression to extract date part with timezone is
{ date: <dateExpression>, timezone: <tzExpression> }
we can either specify the timezone or offset while getting the date parts.
see my answer posted here
to get date from date with timezone America/Chicago
{ $month: {
date: new Date(),
timezone: "America/Chicago"
} }
or with offset
{ $month: {
date: ISODate(),
timezone: "-0500"
} }
Let consider your document contains ISODate as below :
db.collection.insert({"date":new Date()})
Above query insert date in ISODate format now you want to convert this ISODate into give timeZone.
Suppose you want to convert above date to Eastern Daylight Saving Time ( EDT ) epoch time zone conertor then offset converted as 14400 * 1000. First convert ISODate to timeStamp and then use substract EDT OffsetintimeStampand then converttimeStamptoISODate` again.
Check below aggregation query :
db.collection.aggregate({
"$project": {
"timestamp": { //convert ISODate tom timestamp
"$subtract": [{
"$divide": [{
"$subtract": ["$date", new Date("1970-01-01")]
}, 1000]
}, {
"$mod": [{
"$divide": [{
"$subtract": ["$date", new Date("1970-01-01")]
}, 1000]
}, 1]
}]
}
}
}, {
"$project": {
"timeZoneTimeStamp": {
"$subtract": [{ //substract timestamp to given offset if offset will in postive then replace subtract to add
"$multiply": ["$timestamp", 1000]
}, 14400000]
}
}
}, {
"$project": {
"timeZoneTimeStamp": 1, //converted timeZoneTimeStamp if required
"_id": 0,
"newDate": { // newDate is converted timezone ISODate
"$add": [new Date(0), "$timeZoneTimeStamp"]
}
}
})
NOTE :
In above query conversion from ISODATE to timeStamp ref. here
In case if the dates are not changed and constant e.g. something like created_record_date then whichever timezone data you need it, you should pre-calculate and save (as String) along with the same document so that you don't have to run the huge processing at the runtime which could slow down the execution time. in case you have existing records and you want to store the various different timezone data along with the records, think about running a Map-Reduct job and update the documents separately. (let me know if you need the code for that). However, if this date field can be changed as per the business logic then its wise to calculate at runtime. both techniques have their different use cases and their pros and cons.
-$
If you are using mongoose (probably also works in native driver):
import moment from 'moment-timezone'; // this is needed to use .tz() method
import mongoMoment from 'mongodb-moment';
// Initalize mongodb-moment so you can use moment() object directly in mongo query
mongoMoment(moment);
// Add timezone to your_date
const date = moment(your_date)
.tz("Europe/Zagreb");
// Make $gte/$lte queries with date ...