Reversing date conversion using specific format in MongoDB - mongodb

I convert the date to a string so that I can use it for various grouping related work. However in certain groupings the reverse conversion fails.
db.test.aggregate({
$addFields: {
date: "$date",
grouped_format: {
"$dateToString": {
date: '$date',
format: "%Y-%m",
timezone: 'Europe/London',
}
}
}
}, {
$addFields: {
converted: {
"$dateFromString": {
dateString: '$grouped_format',
format: "%Y-%m", // if i comment this out, it works,
timezone: 'Europe/London',
}
}
}
})
The error I get is
{
"message" : "an incomplete date/time string has been found, with elements missing: '2016-01'",
"ok" : 0,
"code" : 241,
"codeName" : "ConversionFailure",
"name" : "MongoError"
}
If I comment the format under $dateFromString, I get the desired result.
{
"_id" : ObjectId("60ca299f10949130abbe5c8e"),
"date" : ISODate("2014-04-21T00:00:00.000Z"),
"grouped_format" : "2014-04",
"converted" : ISODate("2014-03-31T23:00:00.000Z") // London is one hour ahead due to DST so this is correct. -> start of the month
},
{
"_id" : ObjectId("60ca2aaf10949130abbe5c8f"),
"date" : ISODate("2021-01-02T00:00:00.000Z"),
"grouped_format" : "2021-01",
"converted" : ISODate("2021-01-01T00:00:00.000Z") // London is even with UTC so this is correct. -> start of the month
}
Unfortunately, I want to be able to specify any custom format in $dateToString and have it be converted back.
As an example, if the custom format is "%G-%V"which is the ISO week year and ISO week number, specifying the format in $dateToString and $dateFromString works correctly.
I've tried the following formats and here are the results. As you can see, the No. 4. in the table does not specify the minute but it still works fine. However any variation of the date from No.6. to 8. fails.
If I can't reliably reverse the value then I will need to create a special function that covers all edge cases but to do that I need to understand why this is happening in the first place.
No
Format
Works/Fails
1
'%Y-%m-%dT%H:%M:%S.%L'
Works
2
'%Y-%m-%dT%H:%M'
Works
3
'%Y-%m-%dT%H'
Works
4
'%Y-%m-%d'
Works
5
'%G-%V'
Works
6
'%Y-%m'
Fails
7
'%Y-%m-01'
Fails
8
'%Y-%m-00'
Fails
According to the documentation, the default formatter is "%Y-%m-%dT%H:%M:%S.%LZ" but specifying it explicitly also does not work and errors out. No string or null/undefined value seems to work while converting "%Y-%m" unless you omit the format string entirely in $dateFromString. This makes the behaviour very strange.

Related

Timezone query on MongoDB

I'm building an application wherein users can be located in various timezones, and the queries I run are sensitive to their timezones.
The problem I'm having is that MongoDB seems to be ignoring the timezone on query time!
This is an example of a date field "2019-09-29T23:52:13.495000+13:00", here's the full json:
And this is an example of a query:
{
"at": {
"$gte": "2019-09-29T00:00:00+00:00",
"$lte": "2019-09-30T00:00:00+00:00"
}
}
All of my dates are saved either with +12 or +13 because those are the timezones where my customers are at the moment, essentially what that means for the above query is that I should be seeing some results from the 2019-10-01:00:00:00+13:00 due to the first of October still being the 30th of September in UTC, and I'm not.
I'm new to this and not too far down the rabbit hole so I'm open to refactoring/change of thinking if it will make things easier.
For context, in case it makes a difference I'm using PyMongo, and my MongoDB version is 4.2
EDIT!
I've tried converting the "at" field into date, but the timezone seem to have been suppressed or is not visible
With that I also had to change the way I query
{ "at": {
"$gte": ISODate("2019-09-29T00:00:00+00:00"),
"$lte": ISODate("2019-09-30T00:00:00+00:00")
}
}
Didn't help
MongoDB uses the UTC timezone when it stores date. Using your example date:
> ISODate("2019-09-29T23:52:13.495000+13:00")
ISODate("2019-09-29T10:52:13.495Z")
So the +13:00 timezone is converted to Z (UTC) timezone by ISODate(), and this is the value stored in the database.
> db.test.insert({_id:0, at: ISODate("2019-09-29T23:52:13.495000+13:00")})
> db.test.find()
{ "_id" : 0, "at" : ISODate("2019-09-29T10:52:13.495Z") }
Note that "at" is stored in UTC time instead of +13 as inserted.
Querying using ISODate() as per your example works as expected:
> db.test.find({ "at": {
... "$gte": ISODate("2019-09-29T00:00:00+00:00"),
... "$lte": ISODate("2019-09-30T00:00:00+00:00")
... }
... })
{ "_id" : 0, "at" : ISODate("2019-09-29T10:52:13.495Z") }
The timezone information is not visible due to the visual tool you're using. It is visible in the mongo shell as Z, as in the above example.
The advantage of storing UTC is that you can convert them to any other timezones without worrying about daylight saving time. There are date specific aggregation operators that deals with timezones and convert between them. See Date Expression Operators and https://stackoverflow.com/a/48535842/5619724 for examples.

How to convert a date stored as string in MongoDB to ISODate?

I have a collection in which each document has a time field with value stored as similar to "21-Dec-2017".
I want to convert this to ISODate using projection.
My Query:
db.getCollection('orders').aggregate([{
$project:{time : {$add : new Date("$time")}}
}])
But this is returning me ISODate("1970-01-01T00:00:00.000Z") always.
you can try this,
db.getCollection('orders').aggregate([{
$project: {
time: {
$dateToString: {
format: "%d-%m-%G",
date: new Date("$time")
}
}
}
}
])
there is no any string function to get months name eg.Jan,Feb..Dec.
but you can refer https://docs.mongodb.com/manual/reference/operator/aggregation/dateToString/
to more information.
There is no problem in this ISODate("1970-01-01T00:00:00.000Z") format.
you should store date in ISO format but change format on client side according to you.
Basically you want to show date in dd/mm/yy format.
You can use http://momentjs.com/ to show date according to you.

Offset value before grouping

I have a couple of items of which one field is a UTC-based unix timestamp multiplied by 1000 in order to include milliseconds while keeping it a long (integer) value.
{
"title" : "Merkel 'explains' refugee convention to Trump in phone call",
"iso" : "2017-01-31T04:03:53.807+0000",
"id" : NumberLong(1485835433807)
}
{
"title" : "NASA to Explore an Asteroid Containing Enough Mineral Wealth to Collapse the World Economy",
"iso" : "2017-01-30T23:20:27.327+0000",
"id" : NumberLong(1485818427327)
}
{
"title" : "IMGKit: Python library of HTML to IMG wrapper",
"iso" : "2017-01-30T23:15:39.488+0000",
"id" : NumberLong(1485818139488)
}
the iso field is just a text string to ease debugging, it has no other purpose.
I intend to use the method described in https://stackoverflow.com/a/26550803/277267 to resample the items, to create a summary of items per day, initially just a count if items per day.
The problem is that the timestamp (the "id" field) can't really be used to archieve this, because of the UTC offset. Depending on the location of the user (or the local insertion time, ie 00:30 monday local time vs 23:30 sunday UTC time, if the timezone is +1h) an item would belong either to one day or the other, so the field lacks this information.
Assuming I just want to add an offset to the "id" field, ie by 3600000, which is one hour expressed in milliseconds, before starting to resample the data based on the "id" field, how can I archieve this in the aggregation pipeline?
Is there a way to have a first stage which takes the "id" field value, add 3600000 to that value and store it into an "id_offsetted" field, on which I can then execute the next stages?
Version before 3.4
{$project: {
"title" : 1,
"iso" : 1
"id" : 1,
"id_offsetted" : {$add: ["$id", 3600000]}
} }
Version 3.4 onwards
{$addFields: {
"id_offsetted" : {$add: ["$id", 3600000]}
} }

Elastic(search): Query result date format differs from stored format

got a problem with the elastic date format conversion when I parse the results from a query. So i have a default mapping on a date field as following:
"timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
}
and it is stored as "timestamp":"2015-05-06T08:52:56.387Z"
if I execute a max aggregation on that field I get a long value:
"timestamp_max": {
"value": 1430902071110
}
however I want the value be the same as it is stored. I read that one can specify the format in the aggregation but its not working. I tried:
"aggregations":{
"timestamp_max":{
"max":{
"field":"timestamp",
"format" : "dateOptionalTime"
}
}
}
but this gives a SearchParseException ... SearchParseException[[logstash-2015.05.07][0]: query[ConstantScore(BooleanFilter(+no_cache(timestamp:[1429357190515 TO 1431949190515])))],from[-1],size[-1]: Parse Failure [Unexpected token VALUE_STRING in [timestamp_max].]]; ...
What am I doing wrong?
Best regards,
Jan
You're almost there. You just need to specify the date format using the correct formatting pattern like this:
"aggregations":{
"timestamp_max":{
"max":{
"field":"timestamp",
"format" : "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}
}
}
Please note that this is only working from ES 1.5.0 onwards. See the related issue on the ES github.

Unable to get some info in a subdocument

I am trying to get a value in my mongoDB collection. I would like to get the title of a movie and the sales (nbSold) of this movie for the current month.
Here is how my data are stored :
"_id" : ObjectId("52e6a1aacf0b3b522a8a157a"),
"title" : "Pulp Fiction",
"sales" : [
{
"date" : ISODate("2013-11-01T00:00:00Z"),
"nbSold" : 6
},
{
"date" : ISODate("2013-12-01T00:00:00Z"),
"nbSold" : 2
}
]
I'm using mongoose and this is how I build my query for the december of 2013 :
var query = Movie.find({"title":"Pulp Fiction"}, "title sales.nbSold")
.where("sales.date")
.equals(new Date("2013-12-01"));
However, this is the output that I am receiving :
{ title: 'Pulp Fiction', sales: [ { nbSold: 6 }, { nbSold: 2 } ] }
I would like to have only the title associated with the nbSold of the current month (2 in my case). What is the correct way to do this ?
Thanks a lot for your help.
First off, you should close your where call. You should call .exec(callback) when you're done comparing, and you should be using select instead of equals along with $elemMatch. Try this:
var query = Movie.find({"title":"Pulp Fiction"}, "title sales.nbSold")
.where("sales.date")
.select({ sales: { $elemMatch: new Date("2013-12-01") }})
.exec(function(err, doc) {
return console.dir(doc);
});
You should also have a callback. I ran this on my machine, and it definitely works. I posted code earlier that wasn't quite right, but this does the trick. See the documentation for an example.
Also, I'd be concerned as to how you're seeing if the date matches. If the time is off in your Date object but the date matches, Mongo won't find a match. And I don't know exactly how it works in Mongo, but in JavaScript, you can't compare two Date objects directly for equality as they are both different objects with the same value, so you may come across that problem as well. See this post for an example on how to do it.