Found some peculiar behavior was wondering if anyone could help me to understand it so to avoid similar issues in the future.
Creating a cloudant view I want to return only records with a timestamp of the current day.
I was having a hard time getting it to work and found the difference is in having a space before the end of the if condition.
See below for working and not working
if (new Date(Date.parse(doc.details.timestamp)).setHours(0,0,0,0) === new Date().setHours(0,0,0,0) ){
Works to check the current date against the Cloudant doc date
if (new Date(Date.parse(doc.details.timestamp)).setHours(0,0,0,0) === new Date().setHours(0,0,0,0)){
Does not work to check the date against the Cloudant doc date
Full working view below for context
function (doc) {
if (doc.details.location){
if (new Date(Date.parse(doc.details.timestamp)).setHours(0,0,0,0) === new Date().setHours(0,0,0,0) ){
emit(doc.details.location.toLowerCase(), { "location": doc.details.location.toLowerCase(), "status": doc.details.status, "user": doc.details.username, "time": doc.details.timestamp})
}
}
}
All the best,
Scott.
I suspect this may be to do with the execution date/time of the if statement rather than the space involved.
The map function needs to be deterministic regardless of execution time.
It looks like you are guessing that the map function is running at query time (so new Date would emit today's date). Instead, it runs at indexing time, so the value of new Date is whatever the datetime is when the indexing happens. As indexing runs at a different time to document insertion (sometime between insert and when the view is queried), using any form of value that changes with time will produce unpredictable results.
I suspect therefore the space is co-incidental and instead the output of new Date is changing, and so altering what's emitted into your view.
For your problem -- querying for things "today" -- I think you want to instead emit a key like [location, year, month, day]. Your map function would look like:
function (doc) {
if (doc.details.location) {
var l = doc.details.location.toLowerCase();
var d = Date.parse(doc.details.timestamp);
emit([l, d.getFullYear(), d.getMonth(), d.getDate()], {
"location": l,
"status": doc.details.status,
"user": doc.details.username,
"time": doc.details.timestamp
});
}
}
As JavaScript uses 0-based indexes for the month, to query for all items at location Bristol on today, 2-Feb-2017, you'd use key=["bristol",2017,1,2] to in your query to the view.
Related
So am fetching data with mongoose and i would like to modify the data like apply some date formats. Currently i have
const count = await UserModel.countDocuments();
const rows = await UserModel.find({ name:{$regex: search, $options: 'i'}, status:10 })
.sort([["updated_at", -1]])
.skip(page * perPage)
.limit(perPage)
.exec();
res.json({ count, rows });
The above UserModel is a mongoose model
I would like to modify some of objects like applying date formats before the data is returned while still paginating as above.
Currently i have added the following which works but i have to loop through all rows which will be a performance nighmare for large data.
res.json({ count, rows:rows.map(el=>({...el,created_at:'format date here'})) });
Is there a better option
As much as I understood your question, If you need to apply some date formats before showing data on frontend, you just need to pass the retrieved date in a date-formating library before displaying it, like in JS:
const d = new Date("2015-03-25T12:00:00Z");
However, if you want to get date in formatted form, than you must format it before storing. I hope that answer your question.
I think the warning from #Fabian Strathaus in the comments is an important consideration. I would strongly recommend that the approach you are trying to solve sets you up for success overall as opposed to introducing new pain points elsewhere with your project.
Assuming that you want to do this, an alternative approach is to ask the database to do this directly. More specifically, the $dateToString operator sounds like it could be of use here. This playground example demonstrates the basic behavior by adding a formatted date field which will be returned directly from the database. It takes the following document:
{
_id: 1,
created_at: ISODate("2022-01-15T08:15:39.736Z")
}
We then execute this sample aggregation:
db.collection.aggregate([
{
"$addFields": {
created_at_formatted: {
$dateToString: {
format: "%m/%d/%Y",
date: "$created_at"
}
}
}
}
])
The document that gets returned is:
{
"_id": 1,
"created_at": ISODate("2022-01-15T08:15:39.736Z"),
"created_at_formatted": "01/15/2022"
}
You could make use of this in a variety of ways, such as by creating and querying a view which will automatically create and return this formatted field.
I also want to comment on this statement that you made:
Currently i have added the following which works but i have to loop through all rows which will be a performance nighmare for large data.
It's good to hear that you're thinking about performance upfront. That said, your query includes a query predicate of name:{$regex: search, $options: 'i'}. Unanchored and/or case insensitive regex filters cannot use indexes efficiently. So if your status predicate is not selective, then you may need to take a look at alternative approaches for filtering on name to make sure that the query is performant.
How do I use the setDate() function on a date generated from a cursor?
I am trying to correctly generate a query that returns a cursor with values between the One day before the expiry date and the expiry date itself.
Find below my code:
var currentDate = new Date();
var expiryDate;
var dayBeforeExpiry;
var loopThruCollection = buyList.find({});
loopThruCollection.forEach(function (row) {
console.log (row.expiryDate)
expiryDate = row.expiryDate;
});
// expiryDate minus one day, should equal the day before the expiry date.
dayBeforeExpiry = expiryDate.setDate(expiryDate.getDate() -1);
if(currentDate == dayBeforeExpiry){
return buyList.find({ expiryDate : {'$gte': dayBeforeExpiry, '$lte': expiryDate } }).fetch();
}
else {
return buyList.find({ expiryDate : {'$lte': expiryDate}} ).fetch();
}
The results from the code above is:
Thu Mar 30 2017 16:29:35 GMT+0300 (EAT)
2017-02-22T14:46:33+03:00
2017-03-21T17:17:22+03:00
2017-03-21T17:18:45+03:00
2017-01-30T17:42:21+03:00
2017-02-22T15:10:50+03:00
2017-01-27T21:45:57+03:00
Uncaught TypeError: expiryDate.getDate is not a function(…)
I am not sure how I should go about generating the day before variable from the expiryDate.
Kindly point me in the correction.
Based on your output, it looks like not all your expiryDate fields are actually stored as ISODate in mongodb (since they are outputting differently). If they are not stored as ISODate then they won't get mapped automatically to a Date object in Meteor. This is most likely why you are getting your .getDate is not a function(…) error.
You can verify this by querying mongodb directly and looking at the stored values for your field. Check and see if some are being stored as strings vs. ISODate. I'm betting some are strings and some are ISODate.
Also, you cannot use equality operators (e.g. ==, ===, !=, !==) to compare the value of javascript dates, because that is actually comparing the objects themselves (not their value) and it will always be false. Per the javascript comparison operators reference docs...
An expression comparing Objects is only true if the operands reference the same Object.
You should compare the value of dates using the .getTime() method instead.
// won't work...will always return false
if (currentDate == dayBeforeExpiry) {
// do stuff
}
// correct way
if (currentDate.getTime() == dayBeforeExpiry.getTime()) {
// do stuff
}
However, even this may not do what you are wanting because the comparison is down to the millisecond. It might make more sense for you to change your logic and compare the year, month, and day only and exclude time altogether.
With that said, I would highly recommend using moment.js in your meteor app to handle all your date logic. It has built in functions to add/subtract units of time and compare dates. I deal with dates a lot in my app and I find using it a life saver.
You can install it in your meteor app via npm (meteor npm install moment --save) or as a meteor package (meteor add momentjs:moment).
MongoDB provides a way to update a date field by the system on update operations: https://docs.mongodb.com/manual/reference/operator/update/currentDate/. Is there any equivalent to this for insert operations?
You may try to do a few things if you do not want to handle this from code (I have executed the code below directly on mongo shell):
If you want to use $currentDate use update with upsert = true:
db.orders.update(
{"_id":ObjectId()},
{
$currentDate: {
createtime: true
}
},
{ upsert: true }
)
It will now generate the objectid on app server instead of date/time (unless you use raw command).
Use new timestamp or date object directly:
db.orders.insert(
"createtime": new Timestamp()
)
The problem with most driver will be then to make sure the new object is created on mondodb server- not on the machine where the code is running. You driver hopefully allows to run raw insert command.
Both will serve the purpose of avoiding time differences/ time sync issue between application server machines.
The $currentDate is an update operator which populates date field with current date through update operation.
To auto populate date field while insertion of new MongoDB document,please try executing following code snippet
var current_date=new Date();
db.collection.insert({datefield:current_date})
In above code snippet the statement
new Date()
creates a new JavaScript Date object which consists of a year, a month, a day, an hour, a minute, a second, and milliseconds
If you want to populate this value when running it in the server side, and are concerned about it being passed by the client, you can add properties to the data object that is being used in the insert statement only when it will be saved. This way, you can guarantee that it will be added every time with the server's date, not the client's:
Client side
...
let data = { info1: 'value1', info2: 'value2'}
someApi.addInfo(data);
...
Server side
function addInfo(data){
...
data['creationDate'] = new Date();
db.collection.insertOne(data);
...
}
Result will be:
{
info1: 'value1',
info2: 'value2',
creationDate: ISODate("2018-09-15T21:42:13.815Z")
}
If you are passing multiple values for insertion (using insertMany) you have to loop over the items and add this property for all of them.
You can also use this approach when updating documents if for some reason you can't use the $currentDate operator, just be sure you are not replacing any existing properties in the data passed to mongodb.
Since mongo 3.6 you can use 'change stream':
https://emptysqua.re/blog/driver-features-for-mongodb-3-6/#change-streams
to use it you need to create a change stream object by the 'watch' query:
def update_ts_by(change):
update_fields = change["updateDescription"]["updatedFields"].keys()
print("update_fields: {}".format(update_fields))
collection = change["ns"]["coll"]
db = change["ns"]["db"]
key = change["documentKey"]
if len(update_fields) == 1 and "update_ts" in update_fields:
pass
else:
client[db][collection].update(key, {"$set": {"update_ts": datetime.now()}})
client = MongoClient("172.17.0.2")
db = client["Data"]
change_stream = db.watch()
for change in change_stream:
print(change)
update_ts_by(change)
Note, to use the change_stream object, your mongodb instance should run as 'replica set'.
It can be done also as a 1-node replica set (almost no change then the standalone use):
Mongo DB - difference between standalone & 1-node replica set
I'm saving data into the bongo as bulk insert. The data that's an array of JSON object contain date, numeric, alphanumeric data all saved as string.
Sample Data
[{
"CARDNO": "661",
"HOLDERNO": "661",
"HOLDERNAME": "S",
"IODATE": "4/1/2012",
"IOTIME": "00:03:27",
"IOGATENO": "01-3",
"IOGATENAME": "FWork",
"IOSTATUS": "Entry",
"DEPARTMENTNO": "1",
"UPDATE_STATUS": "1"
}, {
"CARDNO": "711",
"HOLDERNO": "711",
"HOLDERNAME": "P",
"IODATE": "4/1/2012",
"IOTIME": "04:35:33",
"IOGATENO": "01-7",
"IOGATENAME": "FDWork",
"IOSTATUS": "Exit",
"DEPARTMENTNO": "3",
"UPDATE_STATUS": "1"
}]
My Query
var start = new Date(2012, 4, 15);
var end = new Date(2012, 4, 1);
collection.find({
"IODATE": {
$gte: start,
$lt: end
}
}).toArray(function (err, data) {
if (err) {
console.log(err);
} else {
console.log(data.length)
}
//res.send(data.length);
res.send(JSON.stringify(data));
});
It's not returning result, I think it is because the value of "IODATE" is in string inside db.
How to work around this issue? I may need to do bulk insert since the data can be of 200 million or so.
One last try at this, because you don't have a good record of accepting good advice.
Your date formats as they stand are going to bite you. Even where trying to work around them. Here are the problems:
The format is not lexical. Which means that even with a string comparison operators like $gte, $lte are just not going to work. A lexical date would be "2012-01-04" in "yyyy-mm-dd" format. That would work with the operators.
You could look at $substr (and it's complete lack of documentation, search on SO for real usage) within aggregate but your date format is lacking the double digit form of day and month ie "04/01/2012", so that is going to blow up the positional nature of the operator. Also you would have to transform before any $match which means you blow up any chance of reducing your pipeline input, so you are stuck with not being able to filter your large resultset by date.
It's a horrible case, but there really is no other practical solution to the large data problem here than to convert your dates. Strings in the form that you have just do not cut it. Either, in order of preference convert to:
BSON date
epoch timestamp as long (whatever)
Lexical string representation (as described)
Your main case seems to be filtering, so updating the dataset is the only pratical alternative. Otherwise you are stuck with "paging" results and doing a lot of manual work, that could otherwise be done server side.
I have a simple Meteor collection and I am trying to insert a document that has a momentjs property into it. So I do:
docId = Col.insert({m: moment()});
However, when I try to get this document back with
doc = Col.findOne({_id: docId})
I get "Invalid date" for doc.m like so:
Object {_id: "wnHzTpHHxMSyMxmu3", m: "Invalid date"}
Anyone?!
I strongly recommend storing dates as Date objects and using moment to format them after they are fetched. For example:
Posts.insert({message: 'hello', createdAt: new Date});
Then later when you want to display the date:
var date = Posts.findOne().createdAt;
moment(date).format('MMMM DD, YYYY');
Moments are not designed to be directly serializable. They won't survive a round-trip to/from JSON. The best approach would be to serialize an ISO8601 formatted date, such as with moment().toISOString() or moment().format(). (toISOString is prefered, but will store at UTC instead of local+offset).
Then later, you can parse that string with moment(theString) and do what you want with it from there.
David's answer is also correct. Though it will rely on whatever Metor's default mechanism is for serializing Date objects, as Date also cannot exist directly in JSON. I don't know the specifics of Meteor - but chances are it's either storing an integer timestamp or just using Date.toString(). The ISO8601 format is much better suited for JSON than either of those.
UPDATE
I just took a glance at the docs for Meteor, which explain that they use an invented format called "EJSON". (You probably know this, but it's new to me.)
According to these docs, a Date is serialized as an integer timestamp:
{
"d": {"$date": 1358205756553}
}
So - David's answer is spot on (and should remain the accepted answer). But also, if you are doing something other than just getting the current date/time, then you might want to use moment for that. You can use yourMoment.toDate() and pass that so Meteor will treat it with the $date type in it's EJSON format.
If you would like to have moment objects on find and findOne, save it as a date then transform it on finding it. For example:
Posts = new Mongo.Collection('posts', {
transform: function (doc) {
Object.keys(doc).forEach(field => {
if (doc[field] instanceof Date) {
doc[field] = moment(doc[field]);
}
});
return doc;
}
});
Posts.insert({
title: 'Hello',
createdAt: new Date()
});