I have data that looks like this:
[
{
"start_time" : ISODate("2017-08-22T19:43:41.442Z"),
"end_time" : ISODate("2017-08-22T19:44:22.150Z")
},
{
"start_time" : ISODate("2017-08-22T19:44:08.344Z"),
"end_time" : ISODate("2017-08-22T19:46:25.500Z")
}
]
Is there any way to run an aggregation query that will give me a frequency result like:
{
ISODate("2017-08-22T19:43:00.000Z"): 1,
ISODate("2017-08-22T19:44:00.000Z"): 2,
ISODate("2017-08-22T19:45:00.000Z"): 1,
ISODate("2017-08-22T19:46:00.000Z"): 1
}
Essentially I want to group by minute, with a sum, but the trick is that each record might count toward multiple groups. Additionally, as in the case with 19:45, the date is not explicitly mentioned in the data (it is calculated as being between two other dates).
At first I thought I could do this with a function like $minute. I could group by minute and check to see if the data fits in that range. However, I'm stuck on how that would be accomplished, if that's possible at all. I'm not sure how to turn a single entry into multiple date groups.
You can use below aggregation in 3.6 mongo version.
db.col.aggregate([
{"$addFields":{"date":["$start_time","$end_time"]}},
{"$unwind":"$date"},
{"$group":{
"_id":{
"year":{"$year":"$date"},
"month":{"$month":"$date"},
"day":{"$dayOfMonth":"$date"},
"hour":{"$hour":"$date"},
"minute":{"$minute":"$date"}
},
"count":{"$sum":1}
}},
{"$addFields":{
"_id":{
"$dateFromParts":{
"year":"$_id.year",
"month":"$_id.month",
"day":"$_id.day",
"minute":"$_id.minute"
}
}
}}
])
Related
I have a structure of...
{ _id = object_id,
user: name,
days: { "4/1/2010": {"checked": true},
"4/2/2011": {"checked": false)}
}
I want to get the total number of days across users. If days was an array, I would do something like...
db.collection.aggregate([{"$group": {"_id": null, {"$sum": {"$size": "$days"}}}}])
but that won't work since I can't use size. Anyone have suggestions?
Note: There may be a different number of days missing in the data structure for each user which is why I want to check the count within each user's days
You can use aggregation pipeline with $objectToArray stage to convert days pair into arrays followed by $sum and $size in a $group stage in 3.4.
db.collection.aggregate([
{"$group":{
"_id":null,
"count":{
"$sum":{
"$size":{"$objectToArray":"$days"}
}
}
}}
])
I have startTime and endTime for all records like this:
{
startTime : 21345678
endTime : 31345678
}
I am trying to find number of all the conflicts. For example if there are two records and they overlap the number of conflict is 1. If there are three records and two of them overlap the conflict is 1. If there are three records and all three overlap the conflicts is 3 i.e [(X1, X2), (X1, X3), (X2, X3)]
As an algorithm I am thinking of sorting the data by start time and for each sorted record checking the end time and finding the records with start time less than the end time. This will be O(n2) time. A better approach will be using interval tree and inserting each record into the tree and finding the counts when overlaps occur. This will be O(nlgn) time.
I have not used mongoDB much so what kind of query can I use to achieve something like this?
As you correctly mention, there are different approaches with varying complexity inherent to their execution. This basically covers how they are done and which one you implement actually depends on which your data and use case is best suited to.
Current Range Match
MongoDB 3.6 $lookup
The most simple approach can be employed using the new syntax of the $lookup operator with MongoDB 3.6 that allows a pipeline to be given as the expression to "self join" to the same collection. This can basically query the collection again for any items where the starttime "or" endtime of the current document falls between the same values of any other document, not including the original of course:
db.getCollection('collection').aggregate([
{ "$lookup": {
"from": "collection",
"let": {
"_id": "$_id",
"starttime": "$starttime",
"endtime": "$endtime"
},
"pipeline": [
{ "$match": {
"$expr": {
"$and": [
{ "$ne": [ "$$_id", "$_id" },
{ "$or": [
{ "$and": [
{ "$gte": [ "$$starttime", "$starttime" ] },
{ "$lte": [ "$$starttime", "$endtime" ] }
]},
{ "$and": [
{ "$gte": [ "$$endtime", "$starttime" ] },
{ "$lte": [ "$$endtime", "$endtime" ] }
]}
]},
]
},
"as": "overlaps"
}},
{ "$count": "count" },
]
}},
{ "$match": { "overlaps.0": { "$exists": true } } }
])
The single $lookup performs the "join" on the same collection allowing you to keep the "current document" values for the "_id", "starttime" and "endtime" values respectively via the "let" option of the pipeline stage. These will be available as "local variables" using the $$ prefix in subsequent "pipeline" of the expression.
Within this "sub-pipeline" you use the $match pipeline stage and the $expr query operator, which allows you to evaluate aggregation framework logical expressions as part of the query condition. This allows the comparison between values as it selects new documents matching the conditions.
The conditions simply look for the "processed documents" where the "_id" field is not equal to the "current document", $and where either the "starttime"
$or "endtime" values of the "current document" falls between the same properties of the "processed document". Noting here that these as well as the respective $gte and $lte operators are the "aggregation comparison operators" and not the "query operator" form, as the returned result evaluated by $expr must be boolean in context. This is what the aggregation comparison operators actually do, and it's also the only way to pass in values for comparison.
Since we only want the "count" of the matches, the $count pipeline stage is used to do this. The result of the overall $lookup will be a "single element" array where there was a count, or an "empty array" where there was no match to the conditions.
An alternate case would be to "omit" the $count stage and simply allow the matching documents to return. This allows easy identification, but as an "array embedded within the document" you do need to be mindful of the number of "overlaps" that will be returned as whole documents and that this does not cause a breach of the BSON limit of 16MB. In most cases this should be fine, but for cases where you expect a large number of overlaps for a given document this can be a real case. So it's really something more to be aware of.
The $lookup pipeline stage in this context will "always" return an array in result, even if empty. The name of the output property "merging" into the existing document will be "overlaps" as specified in the "as" property to the $lookup stage.
Following the $lookup, we can then do a simple $match with a regular query expression employing the $exists test for the 0 index value of output array. Where there actually is some content in the array and therefore "overlaps" the condition will be true and the document returned, showing either the count or the documents "overlapping" as per your selection.
Other versions - Queries to "join"
The alternate case where your MongoDB lacks this support is to "join" manually by issuing the same query conditions outlined above for each document examined:
db.getCollection('collection').find().map( d => {
var overlaps = db.getCollection('collection').find({
"_id": { "$ne": d._id },
"$or": [
{ "starttime": { "$gte": d.starttime, "$lte": d.endtime } },
{ "endtime": { "$gte": d.starttime, "$lte": d.endtime } }
]
}).toArray();
return ( overlaps.length !== 0 )
? Object.assign(
d,
{
"overlaps": {
"count": overlaps.length,
"documents": overlaps
}
}
)
: null;
}).filter(e => e != null);
This is essentially the same logic except we actually need to go "back to the database" in order to issue the query to match the overlapping documents. This time it's the "query operators" used to find where the current document values fall between those of the processed document.
Because the results are already returned from the server, there is no BSON limit restriction on adding content to the output. You might have memory restrictions, but that's another issue. Simply put we return the array rather than cursor via .toArray() so we have the matching documents and can simply access the array length to obtain a count. If you don't actually need the documents, then using .count() instead of .find() is far more efficient since there is not the document fetching overhead.
The output is then simply merged with the existing document, where the other important distinction is that since theses are "multiple queries" there is no way of providing the condition that they must "match" something. So this leaves us with considering there will be results where the count ( or array length ) is 0 and all we can do at this time is return a null value which we can later .filter() from the result array. Other methods of iterating the cursor employ the same basic principle of "discarding" results where we do not want them. But nothing stops the query being run on the server and this filtering is "post processing" in some form or the other.
Reducing Complexity
So the above approaches work with the structure as described, but of course the overall complexity requires that for each document you must essentially examine every other document in the collection in order to look for overlaps. Therefore whilst using $lookup allows for some "efficiency" in reduction of transport and response overhead, it still suffers the same problem that you are still essentially comparing each document to everything.
A better solution "where you can make it fit" is to instead store a "hard value"* representative of the interval on each document. For instance we could "presume" that there are solid "booking" periods of one hour within a day for a total of 24 booking periods. This "could" be represented something like:
{ "_id": "A", "booking": [ 10, 11, 12 ] }
{ "_id": "B", "booking": [ 12, 13, 14 ] }
{ "_id": "C", "booking": [ 7, 8 ] }
{ "_id": "D", "booking": [ 9, 10, 11 ] }
With data organized like that where there was a set indicator for the interval the complexity is greatly reduced since it's really just a matter of "grouping" on the interval value from the array within the "booking" property:
db.booking.aggregate([
{ "$unwind": "$booking" },
{ "$group": { "_id": "$booking", "docs": { "$push": "$_id" } } },
{ "$match": { "docs.1": { "$exists": true } } }
])
And the output:
{ "_id" : 10, "docs" : [ "A", "D" ] }
{ "_id" : 11, "docs" : [ "A", "D" ] }
{ "_id" : 12, "docs" : [ "A", "B" ] }
That correctly identifies that for the 10 and 11 intervals both "A" and "D" contain the overlap, whilst "B" and "A" overlap on 12. Other intervals and documents matching are excluded via the same $exists test except this time on the 1 index ( or second array element being present ) in order to see that there was "more than one" document in the grouping, hence indicating an overlap.
This simply employs the $unwind aggregation pipeline stage to "deconstruct/denormalize" the array content so we can access the inner values for grouping. This is exactly what happens in the $group stage where the "key" provided is the booking interval id and the $push operator is used to "collect" data about the current document which was found in that group. The $match is as explained earlier.
This can even be expanded for alternate presentation:
db.booking.aggregate([
{ "$unwind": "$booking" },
{ "$group": { "_id": "$booking", "docs": { "$push": "$_id" } } },
{ "$match": { "docs.1": { "$exists": true } } },
{ "$unwind": "$docs" },
{ "$group": {
"_id": "$docs",
"intervals": { "$push": "$_id" }
}}
])
With output:
{ "_id" : "B", "intervals" : [ 12 ] }
{ "_id" : "D", "intervals" : [ 10, 11 ] }
{ "_id" : "A", "intervals" : [ 10, 11, 12 ] }
It's a simplified demonstration, but where the data you have would allow it for the sort of analysis required then this is the far more efficient approach. So if you can keep the "granularity" to be fixed to "set" intervals which can be commonly recorded on each document, then the analysis and reporting can use the latter approach to quickly and efficiently identify such overlaps.
Essentially, this is how you would implement what you basically mentioned as a "better" approach anyway, and the first being a "slight" improvement over what you originally theorized. See which one actually suits your situation, but this should explain the implementation and the differences.
I want to get the last document for each station with all other fields :
{
"_id" : ObjectId("535f5d074f075c37fff4cc74"),
"station" : "OR",
"t" : 86,
"dt" : ISODate("2014-04-29T08:02:57.165Z")
}
{
"_id" : ObjectId("535f5d114f075c37fff4cc75"),
"station" : "OR",
"t" : 82,
"dt" : ISODate("2014-04-29T08:02:57.165Z")
}
{
"_id" : ObjectId("535f5d364f075c37fff4cc76"),
"station" : "WA",
"t" : 79,
"dt" : ISODate("2014-04-29T08:02:57.165Z")
}
I need to have t and station for the latest dt per station.
With the aggregation framework :
db.temperature.aggregate([{$sort:{"dt":1}},{$group:{"_id":"$station", result:{$last:"$dt"}, t:{$last:"$t"}}}])
returns
{
"result" : [
{
"_id" : "WA",
"result" : ISODate("2014-04-29T08:02:57.165Z"),
"t" : 79
},
{
"_id" : "OR",
"result" : ISODate("2014-04-29T08:02:57.165Z"),
"t" : 82
}
],
"ok" : 1
}
Is this the most efficient way to do that ?
Thanks
To directly answer your question, yes it is the most efficient way. But I do think we need to clarify why this is so.
As was suggested in alternatives, the one thing people are looking at is "sorting" your results before passing to a $group stage and what they are looking at is the "timestamp" value, so you would want to make sure that everything is in "timestamp" order, so hence the form:
db.temperature.aggregate([
{ "$sort": { "station": 1, "dt": -1 } },
{ "$group": {
"_id": "$station",
"result": { "$first":"$dt"}, "t": {"$first":"$t"}
}}
])
And as stated you will of course want an index to reflect that in order to make the sort efficient:
However, and this is the real point. What seems have been overlooked by others ( if not so for yourself ) is that all of this data is likely being inserted already in time order, in that each reading is recorded as added.
So the beauty of this is the the _id field ( with a default ObjectId ) is already in "timestamp" order, as it does itself actually contain a time value and this makes the statement possible:
db.temperature.aggregate([
{ "$group": {
"_id": "$station",
"result": { "$last":"$dt"}, "t": {"$last":"$t"}
}}
])
And it is faster. Why? Well you don't need to select an index ( additional code to invoke) you also don't need to "load" the index in addition to the document.
We already know the documents are in order ( by _id ) so the $last boundaries are perfectly valid. You are scanning everything anyway, and you could also "range" query on the _id values as equally valid for between two dates.
The only real thing to say here, is that in "real world" usage, it might just be more practical for you to $match between ranges of dates when doing this sort of accumulation as opposed to getting the "first" and "last" _id values to define a "range" or something similar in your actual usage.
So where is the proof of this? Well it is fairly easy to reproduce, so I just did so by generating some sample data:
var stations = [
"AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL",
"GA", "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA",
"ME", "MD", "MA", "MI", "MN", "MS", "MO", "MT", "NE",
"NV", "NH", "NJ", "NM", "NY", "NC", "ND", "OH", "OK",
"OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT", "VT",
"VA", "WA", "WV", "WI", "WY"
];
for ( i=0; i<200000; i++ ) {
var station = stations[Math.floor(Math.random()*stations.length)];
var t = Math.floor(Math.random() * ( 96 - 50 + 1 )) +50;
dt = new Date();
db.temperatures.insert({
station: station,
t: t,
dt: dt
});
}
On my hardware (8GB laptop with spinny disk, which is not stellar, but certainly adequate) running each form of the statement clearly shows a notable pause with the version using an index and a sort ( same keys on index as the sort statement). It is only a minor pause, but the difference is significant enough to notice.
Even looking at the explain output ( version 2.6 and up, or actually is there in 2.4.9 though not documented ) you can see the difference in that, though the $sort is optimized out due to the presence of an index, the time taken appears to be with index selection and then loading the indexed entries. Including all fields for a "covered" index query makes no difference.
Also for the record, purely indexing the date and only sorting on the date values gives the same result. Possibly slightly faster, but still slower than the natural index form without the sort.
So as long as you can happily "range" on the first and last _id values, then it is true that using the natural index on the insertion order is actually the most efficient way to do this. Your real world mileage may vary on whether this is practical for you or not and it might simply end up being more convenient to implement the index and sorting on the date.
But if you were happy with using _id ranges or greater than the "last" _id in your query, then perhaps one tweak in order to get the values along with your results so you can in fact store and use that information in successive queries:
db.temperature.aggregate([
// Get documents "greater than" the "highest" _id value found last time
{ "$match": {
"_id": { "$gt": ObjectId("536076603e70a99790b7845d") }
}},
// Do the grouping with addition of the returned field
{ "$group": {
"_id": "$station",
"result": { "$last":"$dt"},
"t": {"$last":"$t"},
"lastDoc": { "$last": "$_id" }
}}
])
And if you were actually "following on" the results like that then you can determine the maximum value of ObjectId from your results and use it in the next query.
Anyhow, have fun playing with that, but again Yes, in this case that query is the fastest way.
An index is all you really need:
db.temperature.ensureIndex({ 'station': 1, 'dt': 1 })
for s in db.temperature.distinct('station'):
db.temperature.find({ station: s }).sort({ dt : -1 }).limit(1)
of course using whatever syntax is actually valid for your language.
Edit: You are correct that a loop like this incurs a round-trip per station, and it's great for a few stations, and not so good for 1000. You do still want the compound index on station+dt, though, and to take advantage of a descending sort:
db.temperature.aggregate([
{ $sort: { station: 1, dt: -1 } },
{ $group: { _id: "$station", result: {$first:"$dt"}, t: {$first:"$t"} } }
])
As far as the aggregation query you've posted, I'd make certain that you have an index on dt:
db.temperature.ensureIndex({'dt': 1 })
This will make certain that the $sort at the beginning of the aggregation pipeline is as efficient as possible.
As to whether or not this is the most efficient way to get this data, vs. a query in a loop, will likely be a function of how many data points you have. In the beginning, with "thousands of stations" and perhaps hundreds of thousands of data points I'd think the aggregation approach will be faster.
However, as you add more and more data an issue is that the aggregation query will continue to touch all the documents. This will get increasingly expensive as you scale up to millions or more documents. One approach for that case would be to add a $limit right after the $sort to limit the total number of documents being considered. That's a bit hacky and inexact but it would help to limit the total number of documents that need to be accessed.
I've got a MongoDB setup and running and I'm issuing a query that looks like this:
db.indexCollection.aggregate([
{$match:
{'term': {$regex: 'gas'},
'term': {$regex: 'carbon'},
'term': {$regex: 'hydro'}}
},
{$sort:
{'documentFrequency': 1,
'postingsList._id.termFrequency': 1}
},
{$group:
{_id: {_id: '$postingsList._id.documentID'}}
}
],
{allowDiskUse: true});
The documentIDs that are returned should contain all of the three words (or subwords) specified in the match expression (logical AND), sorted by document frequency in ascending order (priority to small document frequencies) and term frequency in descending order (priority to documents that make the most use of these terms, thus have the highest term frequency).
However, I notice that when I change the value associated with 'postingsList._id.termFrequency' from 1 to -1, nothing changes.. which makes me think that I'm doing something wrong with the sorting. The order of the output (just a list of documentIDs) does change if I change the value associated with the 'documentFrequency'.
Any ideas?
EDIT:
Sample output format when I execute a simple find query on the key 'term',
{
“_id” : ObjectId(“5941b6ad3de5cb1799f8ea26"),
“term” : “gases”,
“documentFrequency” : 2,
“postingsList” : [
{
“documentID” : 8317982,
“termFrequency” : 4
},
{
“documentID” : 9587169,
“termFrequency” : 1
}
]
}
The output after the $group stage is not as you were hoping, because a $group stage's output may not reflect the order of the input data; to be specific, the documentation says:
$group does not order its output documents.
If you want the output to be ordered after you do a $group, then you need to put a $sort after the $group.
I have a collection like :
{"x":{"y":[
{"date":ISODate("2014-07-24T21:00:00.000Z"),"k":5 },
{"date":ISODate("2014-07-22T21:00:00.000Z"),"k":6 } ] }}
I want to sort y array according to "date" parameter.So i made the code like this :
query.with(new Sort(Sort.Direction.ASC, "y.date"));
I want to output like this:
{"x":{"y":[
{"date":ISODate("2014-07-22T21:00:00.000Z"),"k":5 },
{"date":ISODate("2014-07-24T21:00:00.000Z"),"k":6 } ] }}
How can i make the output like this ? is the code that i made ,it is ok ?
Using aggregation, your query should look like this, since you want to sort elements of 'y' based on 'date' in ascending order.
db.test.aggregate([ {$unwind: "$y"},
{$sort: {"y.date":1}},
{$group: {_id:"$_id", y: {$push:"$y"}}} ]);