My issue is rather specific so I'll try to explain my setup first.
I have a collection called clients, which is a master list of all clients. The model for it is:
{
id: String,
organizationId: Number,
networkId: String,
deviceSerial: String,
}
(irrelevant properties removed)
I also have a collection called clienttransactions, which is a list of when clients have gone online or offline. So each time a client comes online, it adds a record saying it came online (online: true), and vice-versa for when a client goes offline (online: false). The model for that looks like this:
{
clientId: String,
deviceSerial: String,
networkId: String,
organizationId: Number,
ts: Number,
online: Boolean
}
ts is a unix timestamp in seconds. Also if you're wondering why I need all those foreign keys on each record, it's because of the way the API where I get this data from works. So just ignore that.
issue:
Given a deviceSerial, networkId, and organizationId, I want to find all clients that were online at any point between a given time frame (given a start time and end time in epoch seconds).
Possible edge case: There could be times when a client came online before the given start time, and stayed online until after the given end time. In this case, there will be no transaction record within the time frame, but the client should still be seen as online.
Accounting for this case is what I'm having the most trouble with, since I can't simply just search for online transactions between the time frame. If there are no transactions for a client in the time frame, then I need to search outside the time frame to see if the last transaction made before the start time for that client was an online one.
I'm not super well-versed on the aggregation pipeline yet, so this is as far as I got:
const startTime = 1550601742;
const endTime = 1550599341;
ClientTransaction.aggregation([
{
$match: {
organizationId: 600381,
networkId: 'N_651896046061895525',
deviceSerial: 'Q2MN-3CUN-6GQM',
ts: {$lt: endTime}
}
},
{
$group: {
_id: '$clientId',
lastStatus: {
$max: '$ts'
},
online: {
$last: '$online'
}
}
}
]);
I think I'm halfway there with this. It finds all transactions for unique clients before the end time, but stops before process of checking if the client was actually online during the time frame.
You are looking for all clients whose latest activity is an online activity before start time or has online/offline activity between start and end time.
So something like should work
ClientTransaction.aggregation([
{ $match: {
organizationId: 600381,
networkId: 'N_651896046061895525',
deviceSerial: 'Q2MN-3CUN-6GQM',
ts: {$lte: endTime}
}
},
{ $sort:{"clentId":1, "ts":-1 } },
{ $group: {
_id: '$clientId',
latest: {
$first: '$$ROOT'
}
}},
{ $match:{
$or:[
{"latest.online":true,"latest.ts":{$lt:startTime}},
{"latest.ts":{$gte:startTime, $lte:endTime}}
]
}}
]);
Related
Scenario:
I got a entity called "devices", the devices are changing its "assignation", but not many time in its cycle life.
{
device_id: "1234",
...,
assignation: {
status: "assigned",
ts: ...
}
}
Is good idea maintain a log inside object assignation (easy to implement).
{
device_id: "1234",
...,
assignation: {
status: "registered",
ts: 2020-11-04-..,
log: [
["registered", 2020-11-04-..],
["assigned", 2020-10-01-..],
["unassigned", 2020-01-01-..],
]
}
}
Or its better to keep this in another collection? Another collection makes not atomic operation in my insertions (importation process also).
Any other way to do this?
I have a users in MongoDB and each user has an interface allowing them to set their current state of hunger being a combination of "hungry", "not hungry", "famished", "starving", or "full"
Each user can enter a multiple options for any period of time. For example, one use case would be "in the morning, record how my hunger is" and the user can put "not hungry" and "full". They can record how their hunger is at any time in the day, and as many times as they want.
Should I store the data as single entries, and then group the data by a date in MongoDB later on when I need to show it in a UI? Or should I store the data as an array of the options the user selected along with a date?
It depends on your future queries, and you may want to do both. Disk space is cheaper than processing, and it's always best to double your disk space than double your queries.
If you're only going to map by date then you'll want to group all users/states by date. If you're only going to map by user then you'll want to group all dates/states by user. If you're going to query by both, you should just make two Collections to minimize processing. Definitely use an array for the hunger state in either case.
Example structure for date grouping:
{ date: '1494288000',
time-of-day: [
{ am: [
{ user: asdfas, hunger-state: [hungry, full] },
{ user: juhags, hunger-state: [full] }
],
pm: [
{ user: asdfas, hunger-state: [hungry, full] },
{ user: juhags, hunger-state: [full] }
]}]}
It depends on how you are going to access it. If you want to report on a user's last known state, then the array might be better:
{
user_id: '5358e4249611f4a65e3068ab',
timestamp: '2017-05-08T17:30:00.000Z',
hunger: ['HUNGRY','FAMISHED'],
}
The timestamps of multiple records might not align perfectly if you are passing in the output from new Date() (note the second record is 99 ms later):
{
user_id: '5358e4249611f4a65e3068ab',
timestamp: '2017-05-08T17:30:00.000Z',
hunger: 'HUNGRY',
}
{
user_id: '5358e4249611f4a65e3068ab',
timestamp: '2017-05-08T17:30:00.099Z',
hunger: ['FAMISHED',
}
You should probably look at your data model though and try to get a more deterministic state model. Maybe:
{
user_id: '5358e4249611f4a65e3068ab',
timestamp: '2017-05-08T17:30:00.000Z',
isHungry: true,
hunger: 'FAMISHED',
}
Imagine I have a document structure like this.
{
_id: ObjectId('internalId'),
externalId: 'externalId',
history: [
{
effective: ISODate('2000-02-01T00:00:00.000Z'),
property: 'new value'
},
{
effective: ISODate('2000-01-01T00:00:00.000Z'),
property: 'value'
}
]
}
Each time this document is read, all of the properties are merged together in historical order into a final state, possibly stopping at a specific point in time.
To add a new history item, I would need to perform something like this.
{
$push: {
history: {
property: 'even newer value',
effective: new Date()
}
},
$setOnInsert: {
externalId: externalId
}
}
I would like to find a way to make sure that an update that does not modify the actual merged history state is never stored. However it seems like this would require a separate read operation, and thus an (external) pessimistic lock to be held, while it was determined if a revision could proceed.
This feels like an incorrect design. Help!
I have a very certain thing i want to accomplish, and I wanted to make sure it is not possible in mongoose/mongoDB before I go and code the whole thing myself.
I checked mongoose-ttl for nodejs and several forums and didn't find quite what I need.
here it is:
I have a schema with a date field createDate. Now i wish to place a TTL on that field, so far so good, i can do it like so (expiration in 5000 seconds):
createDate: {type: Date, default: Date.now, expires: 5000}
but I would like my users to be able to "up vote" documents they like so those documents will get a longer period of time to live, without changing the other documents in my collection.
So, Can i change a TTL of a SINGLE document somehow once a user tells me he likes that document using mongoose or other existing npm related modules?
thank you
It has been more than a year, but this may be useful for others, so here is my answer:
I was trying accomplish this same thing, in order to allow a grace period after an entry deletion, so the user can cancel the operation afterwards.
As stated by Mike Bennett, you can use a TTL index making documents expire at a specific clock time.
Yo have to create an index, setting the expireAfterSeconds to zero:
db.yourCollection.createIndex({ "expireAt": 1 }, { expireAfterSeconds: 0 });
This will not affect any of the documents in your collection, unless you set expireAfterSeconds on a particular document like so:
db.log_events.insert( {
"expireAt": new Date('July 22, 2013 14:00:00'),
"logEvent": 2,
"logMessage": "Success!"
} )
Example in mongoose
Model
var BeerSchema = new Schema({
name: {
type: String,
unique: true,
required: true
},
description: String,
alcohol: Number,
price: Number,
createdAt: { type: Date, default: Date.now }
expireAt: { type: Date, default: undefined } // you don't need to set this default, but I like it there for semantic clearness
});
BeerSchema.index({ "expireAt": 1 }, { expireAfterSeconds: 0 });
Deletion with grace period
Uses moment for date manipulation
exports.deleteBeer = function(id) {
var deferred = q.defer();
Beer.update(id, { expireAt: moment().add(10, 'seconds') }, function(err, data) {
if(err) {
deferred.reject(err);
} else {
deferred.resolve(data);
}
});
return deferred.promise;
};
Revert deletion
Uses moment for date manipulation
exports.undeleteBeer = function(id) {
var deferred = q.defer();
// Set expireAt to undefined
Beer.update(id, { $unset: { expireAt: 1 }}, function(err, data) {
if(err) {
deferred.reject(err);
} else {
deferred.resolve(data);
}
});
return deferred.promise;
};
You could use the expire at clock time feature in mongodb. You will have to update the expire time each time you want to extend the expiration of a document.
http://docs.mongodb.org/manual/tutorial/expire-data/#expire-documents-at-a-certain-clock-time
Like Facebook, I would like to aggregate the results. But I can't figure out how to go about it.
Example:
Let's say 10 users like my posts.
I don't want to get 10 notifications. 1 is of course enough.
This is my schema:
var eventLogSchema = mongoose.Schema({
//i.e. Somebody commented, sombody liked, etc.
event: String,
//to a comment, to a post, to a profile, etc.
toWhat: String,
//who is the user we need to notify?
toWho: {type: mongoose.Schema.Types.ObjectId, ref:'User'},
//post id, comment id, whatever..
refID: {type: mongoose.Schema.Types.ObjectId},
//who initiated the event.
whoDid: {type: mongoose.Schema.Types.ObjectId, ref: 'User'},
// when the event happened
date: {type:Date, default: Date.now()},
//whether the user already saw this notification or not.
seen: {type:Boolean, default: false}
})
so I need to count the times
Ex.1: event='liked' and toWhat="post" and refID=myPostID and seen=false
But at the same time, I would like to populate the last event with this parameters on the 'who' path so I could display "Michael and 9 other people liked your post(link to post)"
Every way I can think of doing this is clunky and requires multiple queries that feel like they would cost a lot of system resources and I am wondering if there's a simple way to do it.
Actually it gets more complicated then that.
I do not want to specify values like I did in Ex.1.
Instead I would like to say
aggregate all events with similar 'event', 'toWhat',
'refID' with value seen=false and populate the last one on the 'who' path.
Would love some reading materials, links, advice, or anything.
Thanks!
Managed to solve it like this.
Not sure if it's optimal, but it works.
//The name of my Schema
Notification.aggregate([
{
$match: {
//I only want to display new notifications
seen: {$ne: true}
//to query a specific user add
// toWho: UserID
}
},
{
$group: {
//groups when event, toWhat, and refID are similar
_id: {
event: '$event',
toWhat: '$toWhat',
refID: '$refID',
},
//gets the total number of this type of notification
howMany: {$sum: 1},
//gets the date of the last document in this query
date: {$max: '$date'},
//pulls the user ID of the last user in this query
user: {$last: '$whoDid'}
}
}
]).exec(function (err, results) {
if (err) throw err;
if (results) {
//after I get the results, I want to populate my user to get his name.
Notification.populate(results, {path: 'user', model: "User"}, function (err, notifications) {
if (err) throw err;
if (notifications) res.send(notifications);
})
}
})
I'm not sure whether it's possible to populate the aggregated result in one query, I assume that if it's possible it would be optimal, but so far, this seems acceptable for my needs.
Hope this helps.