I created a web-scraper to store data for a week to find a trend.
I wrote code to delete data from more than week ago every time the script runs.
However the data is still being stored for more than a week ago, is there a reason for this?"
example coin data createdAt field looks like
"createdAt": {
"$date": "2021-08-11T10:55:19.843Z"
},
coinSchema.statics.deleteOldData = async function () {
// delete old data
const today = new Date(Date.now());
today.setHours(0, 0, 0, 0);
const oneWeekAgo = new Date(Date.now());
const pastDate = oneWeekAgo.getDate() - 7;
oneWeekAgo.setDate(pastDate);
await this.deleteMany({
createdAt: {
$gte: today,
}, // 16 < 17 wont delete it prevent duplicates for one day
});
await this.deleteMany({
createdAt: {
$lt: pastDate,
}, // from 1 week ago
});
};
in the script i have the this run
async function main() {
await Coin.deleteOldData();
my coin model looks like :
const coinSchema = mongoose.Schema(
{
specNo: {
type: String,
required: true,
},
coinName: {
type: String,
required: true,
},
fullName: {
type: String,
required: false,
},
category: {
type: String,
},
array: [
{
GradeName: String,
PopulationCount: String,
trend: { type: Number, default: 0 },
},
],
},
{
timestamps: true,
}
);
Did you look at TTL based index? https://docs.mongodb.com/manual/core/index-ttl/
This is a good way to cleanup old data where DB itself takes care of it. In your case 7 days is 604800 seconds so if you create a index on createdAt with ttl 604800 then you should be all good!
db.collection.createIndex( { "createdAt ": 1 }, { expireAfterSeconds: 604800 } )
You can do it like this:
db.collection.deleteMany({
createdAt: { $lte: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString() },
})
Related
I need to save daily statistics with the number of message sent using WhatsApp using MongoDB, using mongoose / NodeJS.
According to document, using $inc, if the entry not exist a new is created.
But no one document is created when running this code.
I have this model
const mongoose = require('../database');
const StatWpSchema = new mongoose.Schema({
userId: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
require: true,
},
date: {
type: Date,
require: true,
},
quantity: {
type: Number,
default: 0,
}
});
StatWpSchema.index({
userId: 1,
date: {
index: true,
sparse: true
},
});
const StatWp = mongoose.model('StatWp', StatWpSchema);
module.exports = StatWp;
And the code to save the statistics.
const statWp = require('../models/statWp');
let today = new Date().toISOString().slice(0, 10);
statWp.findOneAndUpdate(
{ userId: _userId, date: today },
{ $inc: { quantity: 1 } },
{ upsert: true },
function (err, response) { }
);
so I have this schema:
{
email: {
type: String,
},
username: { type: String },
createdAt: {
type: Date,
default: Date.now,
},
points: {
type: Number,
default: 0,
},
location: {
type: String,
},
validation: {
type: String,
},
isValidated: {
type: Boolean,
default: false,
},
roles: [
{
type: mongoose.Schema.Types.ObjectId,
ref: "Role",
},
],
password: {
type: String,
},
},
{ collection: "users" }
)
Now that I'm trying to id is to delete this document automatically after 2 minutes if this parameter isValidated is set to false. I know if I set expire_at it will delete document, but don't know how to do this?
I would just set expire_at to 2 minutes in the future in the same place you set isValidated to false.
//Pseudocode because I'm not sure the exact syntax for this
let invalidate = (document) => {
document.isValidated = false
date = new Date()
date += 2 minutes
document.expire_at = date
document.save()
}
Objective:
I would like to update each document's points_left with the document's max_points value.
Player.js (Schema)
import mongoose from 'mongoose';
let Schema = mongoose.Schema;
let PlayerSchema = new Schema({
player_id: {
type: String,
required: true
},
points_left: {
type: Number,
default: 0
},
max_points: {
type: Number,
default: 5
},
created_date:{
type: Date,
default: Date.now
},
});
let Player = mongoose.model("players", PlayerSchema);
export default Player;
cron.js (cron job that plays every 24 hours)
/** This is not actually updating **/
Player.updateMany(
{},
[{"$set": { points_left: "$max_points" }}]
)
The below query works on MongoDB via terminal.
db.players.update(
{},
[{"$set": {points_left: "$max_points"}}],
{ multi : true }
)
Expected: {player_id: 1, points_left: 5, max_points: 5, created_date: 1234567890}
Actual: {player_id: 1, points_left: 0, max_points: 5, created_date: 1234567890}
If your using an async function make sure you are using await before calling updateMany
await Player.updateMany(
{},
[{"$set": { points_left: "$max_points" }}]
);
If not execute the query or it will not run, .then() will also execute if you need a callback.
Player.updateMany(
{},
[{"$set": { points_left: "$max_points" }}]
).exec();
I found a work-around. This one cycles through all of the documents and saves the points_left to max_points
Player.find({})
.then( docs => {
docs.forEach(doc => {
doc.points_left = doc.max_points
doc.save();
})
});
```
Trying to update a dynamic field in a mongoose findAndUpdate, but with no luck.
I have the following schema:
const schema = new Schema({
date: { type: String },
totalVisits: { type: Number, default: 0},
hourStats: Object
});
hourStat is a dynamic object, created by this function:
createHourStatsObject: function () {
const hourObject = {};
for (let i = 0; i < 24; i++) {
hourObject[i] = {
newUsers: 1
}
}
return hourObject;
}
I am trying to write an insertOrUpdate expression, with no luck. ( MongoError: Updating the path 'hourStats' would create a conflict at 'hourStats'
)
const currentHour = currentTime.getHours();
return DailyStatisticsCollecion.findOneAndUpdate({
date: helpers.getTodayDate()
},
{
$setOnInsert: {
date: helpers.getTodayDate(),
hourStats: helpers.createHourStatsObject(),
},
$inc: {
totalVisits: 1,
['hourStats.' + currentHour + '.newUsers']: 1
},
},
{
upsert: true,
setDefaultsOnInsert: true
},
)
How can I increment the hourStat's current hour's totalVisits value otherwise?
Woops, found the answer here.
Basically, I cannot insert and increment the same field in the same query.
So I should do this:
return DailyStatisticsCollecion.findOneAndUpdate({
date: helpers.getTodayDate()
},
{
$setOnInsert: {
date: helpers.getTodayDate(),
},
$inc: {
totalVisits: 1,
['hourStats.' + currentHour + '.newUsers']: 1
},
},
{
upsert: true,
setDefaultsOnInsert: true
},
)
This way the object did not get created though. So if anybody knows a full solution, with which I can create the "emtpy" object AND increment a value if needed, it would be much appreciated.
I'm trying to create a document, that last 120 seconds, and as soon as i call this method i want the TTL to restart.
At the moment i can"t update my document, after 120 sc .. the document get deleted and re-created instead of being always updated.
There is my collection :
LaptopConnections = new Mongo.Collection('laptopConnection');
let LaptopConnectionSchema = new SimpleSchema({
creationDate: {
type: Date,
label: "Date when the laptop was created",
defaultValue: Date.now
},
"state.date": {
type: Date,
label: "time when the laptop was updated",
autoValue: function () {
return new Date;
}
}
}
, { timestamps: true }
)
LaptopConnections.attachSchema(LaptopConnectionSchema)
And there is my method :
Meteor.startup(() => {
LapConnections._ensureIndex({ creationDate: 1 }, { expireAfterSeconds: 120 });// Will delete the collection after ~~ two minutes,
});
Meteor.methods({
create_lapconnection(lap_id) {
check(lap_id, String);
if (!LapConnections.findOne({ _id: lap_id })) {
console.log('workiiing',lap_id)
LaptopConnections.insert({
_id: box_id,
creationDate: Date.now(),
});
} else {
console.log('updated',lap_id)
LaptopConnections.update({ _id: lap_id }, { upsert: true }, {
$set: {
"state.date": Date.now(),
}
}
);
}
}
})
You are updating the state.date field while your index sits on the creationDate field. Once you update creationDate or change your index to use the state.date field instead it should work.