I am getting started with the Changestream in Mongo. In my current setup a stitch functions inserts the changelog events into a revision collection. However when I read data from the collection, I can't convert the Timestamp fields. I have tried with the following 2 attempts:
1) A pipeline
[
{
$match: {
'documentKey._id': _id,
},
},
{
$sort: { _id: -1 },
},
{
$addFields: {
convertedDate: { $toDate: 'clusterTime' },
},
},
]
But it gives the error: Error parsing date string 'clusterTime'; 0: passing a time zone identifier as part of the string is not allowed 'c'; 6: Double timezone specification 'r'
2) The bson Timestamp class
import { Timestamp } from 'bson';
const asTimestampInstance = new Timestamp(v.clusterTime);
But here typescript gives me the error: Expected 2 arguments, but got 1.ts(2554)
index.d.ts(210, 30): An argument for 'high' was not provided.
In Altas, the clustertime correctly looks like a timestamp:
I hope that I am just missing something simple :)
Unfortunately $toDate doesn't work with timestamps directly. At least not in v4.0.
The argument should be either a number, a string, or an ObjectId.
You need to convert Timestamp to string first:
$addFields: {
convertedDate: { $toDate: {$dateToString:{date:"$clusterTime"}} },
},
2) The bson Timestamp class
You should take first 32-bit value from BSON's Timestamp class instance, it means seconds Epoch time, then multiply seconds on 1000 and make it milliseconds Epoch time, then call JS Date constructor.
If v is document from ChangeStream, v.clusterTime is BSON Timestamp class object. So, you should write:
const date = new Date(v.clusterTime.getHighBits() * 1000);
This example worked for me on MongoDB 4.0, ODM Mongoose 5.12, Node.js 12.
Related
I'm trying to match the values greater than and less than a datetime parameter which holds the value for the current date time after it's been formatted with momentjs.
The datetime variable
datetime = moment.utc().format('YYYY-MM-DDTHH:mm:ssZ')
The values I'm trying to match exist in a mongo's document as seen here in the picture
The code I used for matching :
if (_params.datetime) {
_mongoParams.push({
$and: [
{
_rewards: {
_endDatetime: { $lte: new Date(datetime) }
},
},
{
_rewards: {
_startDatetime: { $gte: new Date(datetime) }
},
}
],
})
}
The problem is I'm not getting any results when I try to make requests with the datetime parameter. I tried placing the datetime variable inside an ISODate constructor but it didn't work too.
How can I possibly match the dates ?
In Database the date format stored is as below. Adding two same fields but values are different.
packageDeliveredTime : 2020-08-21 2:39:37
packageDeliveredTime : 2020-08-21 09:3:45
Due to the the above format few of the API's and database query's are not filtering the data correctly and like this there is so many records that got stored in the data base.
The above date format needs to get updated as below.
packageDeliveredTime : 2020-08-21 02:39:37
packageDeliveredTime : 2020-08-21 09:03:45
You should never store date/time values as string, it's a design flaw. Store always proper Date objects.
In order to convert the strings to date function $dateFromString is not sufficient. You can use moment.js library to do this:
db.collection.find({ packageDeliveredTime: { $exists: true, $type: "string" } }).forEach(function (doc) {
db.collection.updateOne(
{ _id: doc._id },
{ $set: { packageDeliveredTime: moment(doc.packageDeliveredTime).toDate() } }
);
})
we're trying to remove documents older than 3 months in a specific collection.
There's no TTL configured on this collection and no single date/time attribute on those documents.
How can I remove those old documents anyway? is there any script I could run to make it automatically?
thanks
Assuming you did not generate your own _id field the ObjectId contains a timestamp, from the docs:
The 12-byte ObjectId value consists of: ...
a 4-byte timestamp value, representing the ObjectId's creation, measured in seconds since the Unix epoch
So if you're using Mongo version 4.0+ you could use $toDate, match the documents and overwrite the current collection using $out
db.collection.aggregate([
{
$addFields: {
shouldKeep: {
$lt: [
{
$subtract: [
"$$NOW",
{
$toDate: "$_id"
}
]
},
7776000// 90 days in seconds
]
}
}
},
{
$match: {
shouldKeep: true
}
},
{
$project: {
shouldKeep: 0
}
},
{
out: "curr_collection"
}
])
Mongo Playground
Mind you this is a POC example however this does not deal with many issues, such as timezones. exact month start and ending ( currently it calculates 90 days back ) and more.
Not mentioning that using $out on a large collection contains a lot of overhead.
My recommendation is to paginate the results and do this in code.
For nodejs for example you can use ObjectId's getTimestamp method, like so: (pseudo code)
const someDocuments = [...];
for each document:
const timestamp = document._id.getTimestamp();
if (timestamp < 3 months ago) delete document.
Now in code you can handle timezones, month start dates and scale issues relatively easily.
I want to obtain the records that the "FECHA_FIN" field is greater than or equal to today's date.
this is an example of my data:
but with this query:
db.getCollection('susp_programadas').find( {"FECHA_FIN":{ $gte: new Date("YYYY-mm-dd") }} )
I do not get results, what am I doing wrong? Thank you
You can convert the date to an ISO date and query that way. Since you stored the date as a string mongo has no idea how to query it against an ISO date without conversion.
If you stored your date in mongo as the default ISO date then you could have easily done this:
db.getCollection('susp_programadas').find({"FECHA_FIN":{$gte: new Date()}})
So this is how you can do it now:
db.getCollection('susp_programadas').aggregate([
{
$project: {
date: { $dateFromString: { dateString: '$FECHA_FIN' }}
}
},
{ $match: { date: { $gte: new Date() }}}
])
You can use the $dateFromString in an aggregate query with a $match to get the results you want. Note that $dateFromString is only available in MongoDB version 3.6 and up.
If there is no way to convert your data to ISODate or upgrade your DB you could also consider another solution which via $where:
db.getCollection('susp_programadas').find({
$where: function(){
return new Date(this.FECHA_FIN) > Date.now()
}
})
However that solution suffers from the fact that $where can not use indexes so have that in mind.
So, I need to extract the day-of-week for some objects to make some aggregations. But all my documents have is a timestamp, not a Date. So I'm trying to use $dayOfMonth (and others) with an expression, I can't figure out what it is not working.
Here is my query (along with a helper function to create my date from the timestamp):
db.Logging.aggregate([
{
$match: {
"timestamp": { $gte: dateToTimestamp("2017-04-10") }
}
},
{
$project: {
_id: 0,
timestamp: "$timestamp",
dia: { $dayOfMonth: myDate("$timestamp") }
}
}
])
function dateToTimestamp(str) {
let d = new Date( str );
return d.getTime() + d.getTimezoneOffset()*60*1000;
}
function myDate( ts) {
var d = new ISODate();
d.setTime( ts );
return d;
}
The problem seems to be in passing the value of $timestamp to the myDate function. If I use a literal (e.g. 1492430243) either as the value of ts inside the function or as the value of the parameter passed to myDate it works fine.
In other words: this $dayOfMonth: myDate("1492430243") works.
Although a solution has been shown to work here (Mongodb aggregation by day based on unix timestamp - a pretty ugly solution, if I may add), I want to know why my solution doesn't. As per mongodb docs, $dayOfMonth works with Date types, my function returns a date, so what is wrong?