I'm trying to match the values greater than and less than a datetime parameter which holds the value for the current date time after it's been formatted with momentjs.
The datetime variable
datetime = moment.utc().format('YYYY-MM-DDTHH:mm:ssZ')
The values I'm trying to match exist in a mongo's document as seen here in the picture
The code I used for matching :
if (_params.datetime) {
_mongoParams.push({
$and: [
{
_rewards: {
_endDatetime: { $lte: new Date(datetime) }
},
},
{
_rewards: {
_startDatetime: { $gte: new Date(datetime) }
},
}
],
})
}
The problem is I'm not getting any results when I try to make requests with the datetime parameter. I tried placing the datetime variable inside an ISODate constructor but it didn't work too.
How can I possibly match the dates ?
Related
I have a collection where the same column has Date and String format dates for different records, with the same ISO pattern like 2021-04-22T14:10:48.751779Z.
And I need to build a unified query for search, with such =, >, < options for search.
The second problem is an input search query has a different pattern ("2021-04-22")
So the question is:
can I write some kind of transformation of existing table data before the search, for example from Date to String, or in case I have a string value this transformation would not cause any exception.
In this case, I would be able to perform a search by $eq, $lt, $gt, and $regex query options.
I`m not considering casting strings to dates even if this is possible because my input did not have "14:10:48.751779Z" this part of the date and query would not find anything because of dates would not match. For this case, $regex looks like the only solution.
Or your any other suggestions would be considered.
Current query which not satisfies case when DB column is String type:
{
"aggregate":"collection_name",
"pipeline":[
{
"$match":{
"$and":[
{
"some_column":{
"$eq":"some_value"
}
},
{
"date_column_with_string_or_date_type":{
"$gt":{
"$date":"1980-01-01T00:00:00Z"
}
}
}
]
}
},
{
"$project":{
"_id":1
}
}
]
}
Solved by using $expr and $toString
This works with Date And String types of "date_column"
$match: {
$expr: {
$gt : [
{ $toString:"$date_column" },
"2021-04-22"
]
}
}
operator:
I am getting started with the Changestream in Mongo. In my current setup a stitch functions inserts the changelog events into a revision collection. However when I read data from the collection, I can't convert the Timestamp fields. I have tried with the following 2 attempts:
1) A pipeline
[
{
$match: {
'documentKey._id': _id,
},
},
{
$sort: { _id: -1 },
},
{
$addFields: {
convertedDate: { $toDate: 'clusterTime' },
},
},
]
But it gives the error: Error parsing date string 'clusterTime'; 0: passing a time zone identifier as part of the string is not allowed 'c'; 6: Double timezone specification 'r'
2) The bson Timestamp class
import { Timestamp } from 'bson';
const asTimestampInstance = new Timestamp(v.clusterTime);
But here typescript gives me the error: Expected 2 arguments, but got 1.ts(2554)
index.d.ts(210, 30): An argument for 'high' was not provided.
In Altas, the clustertime correctly looks like a timestamp:
I hope that I am just missing something simple :)
Unfortunately $toDate doesn't work with timestamps directly. At least not in v4.0.
The argument should be either a number, a string, or an ObjectId.
You need to convert Timestamp to string first:
$addFields: {
convertedDate: { $toDate: {$dateToString:{date:"$clusterTime"}} },
},
2) The bson Timestamp class
You should take first 32-bit value from BSON's Timestamp class instance, it means seconds Epoch time, then multiply seconds on 1000 and make it milliseconds Epoch time, then call JS Date constructor.
If v is document from ChangeStream, v.clusterTime is BSON Timestamp class object. So, you should write:
const date = new Date(v.clusterTime.getHighBits() * 1000);
This example worked for me on MongoDB 4.0, ODM Mongoose 5.12, Node.js 12.
So, I need to extract the day-of-week for some objects to make some aggregations. But all my documents have is a timestamp, not a Date. So I'm trying to use $dayOfMonth (and others) with an expression, I can't figure out what it is not working.
Here is my query (along with a helper function to create my date from the timestamp):
db.Logging.aggregate([
{
$match: {
"timestamp": { $gte: dateToTimestamp("2017-04-10") }
}
},
{
$project: {
_id: 0,
timestamp: "$timestamp",
dia: { $dayOfMonth: myDate("$timestamp") }
}
}
])
function dateToTimestamp(str) {
let d = new Date( str );
return d.getTime() + d.getTimezoneOffset()*60*1000;
}
function myDate( ts) {
var d = new ISODate();
d.setTime( ts );
return d;
}
The problem seems to be in passing the value of $timestamp to the myDate function. If I use a literal (e.g. 1492430243) either as the value of ts inside the function or as the value of the parameter passed to myDate it works fine.
In other words: this $dayOfMonth: myDate("1492430243") works.
Although a solution has been shown to work here (Mongodb aggregation by day based on unix timestamp - a pretty ugly solution, if I may add), I want to know why my solution doesn't. As per mongodb docs, $dayOfMonth works with Date types, my function returns a date, so what is wrong?
I currently have a collection with documents like the following:
{ foo: 'bar', timeCreated: ISODate("2012-06-28T06:51:48.374Z") }
I would now like to add a timestampCreated key to the documents in this collection, to make querying by time easier.
I was able to add the new column with an update and $set operation, and set the timestamp value but I appears to be setting the current timestamp using this:
db.reports.update({}, {
$set : {
timestampCreated : new Timestamp(new Date('$.timeCreated'), 0)
}
}, false, true);
I however have not been able to figure out a way to add this column and set it's value to the timestamp of the existing 'timeCreated' field.
Do a find for all the documents, limiting to just the id and timeCreated fields. Then loop over that and generate the timestampCreated value, and do an update on each.
Use updateMany() which can accept aggregate pipelines (starting from MongoDB 4.2) and thus take advantage of the $toLong operator which converts a Date into the number of milliseconds since the epoch.
Also use the $type query in the update filter to limit only documents with the timeCreated field and of Date type:
db.reports.updateMany(
{ 'timeCreated': {
'$exists': true,
'$type': 9
} },
[
{ '$set': {
'timestampCreated': { '$toLong': '$timeCreated' }
} }
]
)
Is there a way to convert string to date using custom format using mongodb shell
I am trying to convert "21/May/2012:16:35:33 -0400" to date,
Is there a way to pass DateFormatter or something to
Date.parse(...) or ISODate(....) method?
Using MongoDB 4.0 and newer
The $toDate operator will convert the value to a date. If the value cannot be converted to a date, $toDate errors. If the value is null or missing, $toDate returns null:
You can use it within an aggregate pipeline as follows:
db.collection.aggregate([
{ "$addFields": {
"created_at": {
"$toDate": "$created_at"
}
} }
])
The above is equivalent to using the $convert operator as follows:
db.collection.aggregate([
{ "$addFields": {
"created_at": {
"$convert": {
"input": "$created_at",
"to": "date"
}
}
} }
])
Using MongoDB 3.6 and newer
You cab also use the $dateFromString operator which converts the date/time string to a date object and has options for specifying the date format as well as the timezone:
db.collection.aggregate([
{ "$addFields": {
"created_at": {
"$dateFromString": {
"dateString": "$created_at",
"format": "%m-%d-%Y" /* <-- option available only in version 4.0. and newer */
}
}
} }
])
Using MongoDB versions >= 2.6 and < 3.2
If MongoDB version does not have the native operators that do the conversion, you would need to manually iterate the cursor returned by the find() method by either using the forEach() method
or the cursor method next() to access the documents. Withing the loop, convert the field to an ISODate object and then update the field using the $set operator, as in the following example where the field is called created_at and currently holds the date in string format:
var cursor = db.collection.find({"created_at": {"$exists": true, "$type": 2 }});
while (cursor.hasNext()) {
var doc = cursor.next();
db.collection.update(
{"_id" : doc._id},
{"$set" : {"created_at" : new ISODate(doc.created_at)}}
)
};
For improved performance especially when dealing with large collections, take advantage of using the Bulk API for bulk updates as you will be sending the operations to the server in batches of say 1000 which gives you a better performance as you are not sending every request to the server, just once in every 1000 requests.
The following demonstrates this approach, the first example uses the Bulk API available in MongoDB versions >= 2.6 and < 3.2. It updates all
the documents in the collection by changing the created_at fields to date fields:
var bulk = db.collection.initializeUnorderedBulkOp(),
counter = 0;
db.collection.find({"created_at": {"$exists": true, "$type": 2 }}).forEach(function (doc) {
var newDate = new ISODate(doc.created_at);
bulk.find({ "_id": doc._id }).updateOne({
"$set": { "created_at": newDate}
});
counter++;
if (counter % 1000 == 0) {
bulk.execute(); // Execute per 1000 operations and re-initialize every 1000 update statements
bulk = db.collection.initializeUnorderedBulkOp();
}
})
// Clean up remaining operations in queue
if (counter % 1000 != 0) { bulk.execute(); }
Using MongoDB 3.2
The next example applies to the new MongoDB version 3.2 which has since deprecated the Bulk API and provided a newer set of apis using bulkWrite():
var bulkOps = [],
cursor = db.collection.find({"created_at": {"$exists": true, "$type": 2 }});
cursor.forEach(function (doc) {
var newDate = new ISODate(doc.created_at);
bulkOps.push(
{
"updateOne": {
"filter": { "_id": doc._id } ,
"update": { "$set": { "created_at": newDate } }
}
}
);
if (bulkOps.length === 500) {
db.collection.bulkWrite(bulkOps);
bulkOps = [];
}
});
if (bulkOps.length > 0) db.collection.bulkWrite(bulkOps);
In my case I have succeed with the following solution for converting field ClockInTime from ClockTime collection from string to Date type:
db.ClockTime.find().forEach(function(doc) {
doc.ClockInTime=new Date(doc.ClockInTime);
db.ClockTime.save(doc);
})
You can use the javascript in the second link provided by Ravi Khakhkhar or you are going to have to perform some string manipulation to convert your orginal string (as some of the special characters in your original format aren't being recognised as valid delimeters) but once you do that, you can use "new"
training:PRIMARY> Date()
Fri Jun 08 2012 13:53:03 GMT+0100 (IST)
training:PRIMARY> new Date()
ISODate("2012-06-08T12:53:06.831Z")
training:PRIMARY> var start = new Date("21/May/2012:16:35:33 -0400") => doesn't work
training:PRIMARY> start
ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ")
training:PRIMARY> var start = new Date("21 May 2012:16:35:33 -0400") => doesn't work
training:PRIMARY> start
ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ")
training:PRIMARY> var start = new Date("21 May 2012 16:35:33 -0400") => works
training:PRIMARY> start
ISODate("2012-05-21T20:35:33Z")
Here's some links that you may find useful (regarding modification of the data within the mongo shell) -
http://cookbook.mongodb.org/patterns/date_range/
http://www.mongodb.org/display/DOCS/Dates
http://www.mongodb.org/display/DOCS/Overview+-+The+MongoDB+Interactive+Shell
I had some strings in the MongoDB Stored wich had to be reformated to a proper and valid dateTime field in the mongodb.
here is my code for the special date format: "2014-03-12T09:14:19.5303017+01:00"
but you can easyly take this idea and write your own regex to parse the date formats:
// format: "2014-03-12T09:14:19.5303017+01:00"
var myregexp = /(....)-(..)-(..)T(..):(..):(..)\.(.+)([\+-])(..)/;
db.Product.find().forEach(function(doc) {
var matches = myregexp.exec(doc.metadata.insertTime);
if myregexp.test(doc.metadata.insertTime)) {
var offset = matches[9] * (matches[8] == "+" ? 1 : -1);
var hours = matches[4]-(-offset)+1
var date = new Date(matches[1], matches[2]-1, matches[3],hours, matches[5], matches[6], matches[7] / 10000.0)
db.Product.update({_id : doc._id}, {$set : {"metadata.insertTime" : date}})
print("succsessfully updated");
} else {
print("not updated");
}
})
How about using a library like momentjs by writing a script like this:
[install_moment.js]
function get_moment(){
// shim to get UMD module to load as CommonJS
var module = {exports:{}};
/*
copy your favorite UMD module (i.e. moment.js) here
*/
return module.exports
}
//load the module generator into the stored procedures:
db.system.js.save( {
_id:"get_moment",
value: get_moment,
});
Then load the script at the command line like so:
> mongo install_moment.js
Finally, in your next mongo session, use it like so:
// LOAD STORED PROCEDURES
db.loadServerScripts();
// GET THE MOMENT MODULE
var moment = get_moment();
// parse a date-time string
var a = moment("23 Feb 1997 at 3:23 pm","DD MMM YYYY [at] hh:mm a");
// reformat the string as you wish:
a.format("[The] DDD['th day of] YYYY"): //"The 54'th day of 1997"