Related
I have array of dates childrendob like
[
"2000-01-21T15:18:49+05:30",
"2008-03-12T15:18:49+05:30"
]
I want to fetch data from users table in which the children birthday is after 7 days from today.
Like suppose today is 15 Jan and result should be all users for which childrendob is 21-1
I need mongodb query for this
Get month and day after 7 days from your client side,
let date = new Date();
date.setDate(date.getDate() + 7); // add 7 days in current date
let month = date.getMonth() + 1; // month
let day = date.getDate(); // date
$filter to iterate loop of dates array, check $and condition from month and day,
$toDate to convert string date to ISO date, it its already in ISO date then you can skip/remove this conversation
$month to get month from date
$dayOfMonth to get date form date
$match for dates should not be empty
db.collection.aggregate([
{
$project: {
dates: {
$filter: {
input: "$dates",
cond: {
$and: [
{ $eq: [{ $month: { $toDate: "$$this" } }, month] }, // input month variable
{ $eq: [{ $dayOfMonth: { $toDate: "$$this" } }, day] } // input day variable
]
}
}
}
}
},
{ $match: { dates: { $ne: [] } } }
])
Playground
I'm trying to build a query that return a total count of users and a count of users created on last week.
There's a field called timeStamp that represents the date of creation.
I'm trying to do this with aggregation, I guess I should first group all users by timeStamp but then I don't know exactly what could I do to achieve this.
EDIT:
Sample user document:
{
"_id" : ObjectId("57be35d6fab7762415376b1b"),
"provider" : "local",
"isValidAccount" : true,
"isActive" : true,
"timeStamp" : ISODate("2016-08-25T00:03:34.533Z"),
"scope" : "getm-user",
"tkbSponsor" : "example#example.com",
"userId" : "example#example.com",
"passwd" : "$2a$14$WARJLD4RtYOApJvTNwQHluLvWpZzQzvUxudIln.j5aQJaxYsJtHEG",
"posFavorites" : [ ],
}
What I do need is a count of ALL users and another count of all users created 7 days ago.
You first need to create a date range query that satisfy the given condition of users created last week, this means defining two variables that will hold the date objects representing the start of the day last week and the end. You will need this to query your collection later on in the pipeline.
You can start with the $group pipeline step that groups all the documents in the collection and calculates the total docs using $sum. You can also calculate the conditional sum based on the date range using the $cond tenary operator to feed the $sum.
The following explains the above approach:
var today = new Date();
var lastWeekStart = new Date(today.getFullYear(), today.getMonth(), today.getDate() - 7);
var lastWeekEnd = new Date(today.getFullYear(), today.getMonth(), today.getDate() - 7);
var start = new Date(lastWeekStart.setHours(0,0,0,0));
var end = new Date(lastWeekEnd.setHours(23,59,59,999));
db.collection.aggregate([
{
"$group": {
"_id": null,
"total": { "$sum": 1 },
"usersCreatedLastWeek": {
"$sum": {
"$cond": [
{
"$and": [
{ "$gte": [ "$timeStamp", start ] },
{ "$lte": [ "$timeStamp", end ] }
]
},
1,
0
]
}
}
}
}
])
Users created last week: timeStamps with week equal to last calendar week.
After declaring today's date. we can use aggregation stages in a pipeline like so:
Project to get time stamp's year and week, and also current year and week.
Project again to compare :
current year with time stamp's year.
current week(-1) with time stamp's week.
Match comparison fields with 0, as 0 means equal.
Lastly group to get total of such time stamps of last week.
Execute this on mongo shell :
var today = new Date();
db.yourCollectionName.aggregate([{
$project: {
"tsYear": {$year: "$timeStamp"},
"tsWeek": {$week: "$timeStamp"},
"todYear": {$year: today},
"todWeek": {$week: today}
}
}, {
$project: {
cmpWeek: {$cmp: ['$tsWeek', {$add: [-1, '$todWeek']}]},
cmpYear: {$cmp: ['$tsYear', '$todYear']}
}
}, {
$match: {
cmpWeek: 0,
cmpYear: 0
}
}, {
$group: {
_id: "UsersCreated",
totalLastWeek: {$sum: 1}
}
}
])
I am working on a project in which I am tracking number of clicks on a topic.
I am using mongodb and I have to group number of click by date( i want to group data for 15 days).
I am having data store in following format in mongodb
{
"_id" : ObjectId("4d663451d1e7242c4b68e000"),
"date" : "Mon Dec 27 2010 18:51:22 GMT+0000 (UTC)",
"topic" : "abc",
"time" : "18:51:22"
}
{
"_id" : ObjectId("4d6634514cb5cb2c4b69e000"),
"date" : "Mon Dec 27 2010 18:51:23 GMT+0000 (UTC)",
"topic" : "bce",
"time" : "18:51:23"
}
i want to group number of clicks on topic:abc by days(for 15 days)..i know how to group that but how can I group by date which are stored in my database
I am looking for result in following format
[
{
"date" : "date in log",
"click" : 9
},
{
"date" : "date in log",
"click" : 19
},
]
I have written code but it will work only if date are in string (code is here http://pastebin.com/2wm1n1ix)
...please guide me how do I group it
New answer using Mongo aggregation framework
After this question was asked and answered, 10gen released Mongodb version 2.2 with an aggregation framework, which is now the better way to do this sort of query. This query is a little challenging because you want to group by date and the values stored are timestamps, so you have to do something to convert the timestamps to dates that match. For the purposes of example I will just write a query that gets the right counts.
db.col.aggregate(
{ $group: { _id: { $dayOfYear: "$date"},
click: { $sum: 1 } } }
)
This will return something like:
[
{
"_id" : 144,
"click" : 165
},
{
"_id" : 275,
"click" : 12
}
]
You need to use $match to limit the query to the date range you are interested in and $project to rename _id to date. How you convert the day of year back to a date is left as an exercise for the reader. :-)
10gen has a handy SQL to Mongo Aggregation conversion chart worth bookmarking. There is also a specific article on date aggregation operators.
Getting a little fancier, you can use:
db.col.aggregate([
{ $group: {
_id: {
$add: [
{ $dayOfYear: "$date"},
{ $multiply:
[400, {$year: "$date"}]
}
]},
click: { $sum: 1 },
first: {$min: "$date"}
}
},
{ $sort: {_id: -1} },
{ $limit: 15 },
{ $project: { date: "$first", click: 1, _id: 0} }
])
which will get you the latest 15 days and return some datetime within each day in the date field. For example:
[
{
"click" : 431,
"date" : ISODate("2013-05-11T02:33:45.526Z")
},
{
"click" : 702,
"date" : ISODate("2013-05-08T02:11:00.503Z")
},
...
{
"click" : 814,
"date" : ISODate("2013-04-25T00:41:45.046Z")
}
]
There are already many answers to this question, but I wasn't happy with any of them. MongoDB has improved over the years, and there are now easier ways to do it. The answer by Jonas Tomanga gets it right, but is a bit too complex.
If you are using MongoDB 3.0 or later, here's how you can group by date. I start with the $match aggregation because the author also asked how to limit the results.
db.yourCollection.aggregate([
{ $match: { date: { $gte: ISODate("2019-05-01") } } },
{ $group: { _id: { $dateToString: { format: "%Y-%m-%d", date: "$date"} }, count: { $sum: 1 } } },
{ $sort: { _id: 1} }
])
To fetch data group by date in mongodb
db.getCollection('supportIssuesChat').aggregate([
{
$group : {
_id :{ $dateToString: { format: "%Y-%m-%d", date: "$createdAt"} },
list: { $push: "$$ROOT" },
count: { $sum: 1 }
}
}
])
Late answer, but for the record (for anyone else that comes to this page): You'll need to use the 'keyf' argument instead of 'key', since your key is actually going to be a function of the date on the event (i.e. the "day" extracted from the date) and not the date itself. This should do what you're looking for:
db.coll.group(
{
keyf: function(doc) {
var date = new Date(doc.date);
var dateKey = (date.getMonth()+1)+"/"+date.getDate()+"/"+date.getFullYear()+'';
return {'day':dateKey};
},
cond: {topic:"abc"},
initial: {count:0},
reduce: function(obj, prev) {prev.count++;}
});
For more information, take a look at MongoDB's doc page on aggregation and group: http://www.mongodb.org/display/DOCS/Aggregation#Aggregation-Group
This can help
return new Promise(function(resolve, reject) {
db.doc.aggregate(
[
{ $match: {} },
{ $group: { _id: { $dateToString: { format: "%Y-%m-%d", date: "$date" } }, count: { $sum: 1 } } },
{ $sort: { _id: 1 } }
]
).then(doc => {
/* if you need a date object */
doc.forEach(function(value, index) {
doc[index]._id = new Date(value._id);
}, this);
resolve(doc);
}).catch(reject);
}
Haven't worked that much with MongoDB yet, so I am not completely sure. But aren't you able to use full Javascript?
So you could parse your date with Javascript Date class, create your date for the day out of it and set as key into an "out" property. And always add one if the key already exists, otherwise create it new with value = 1 (first click). Below is your code with adapted reduce function (untested code!):
db.coll.group(
{
key:{'date':true},
initial: {retVal: {}},
reduce: function(doc, prev){
var date = new Date(doc.date);
var dateKey = date.getFullYear()+''+date.getMonth()+''+date.getDate();
(typeof prev.retVal[dateKey] != 'undefined') ? prev.retVal[dateKey] += 1 : prev.retVal[dateKey] = 1;
},
cond: {topic:"abc"}
}
)
thanks for #mindthief, your answer help solve my problem today. The function below can group by day a little more easier, hope can help the others.
/**
* group by day
* #param query document {key1:123,key2:456}
*/
var count_by_day = function(query){
return db.action.group(
{
keyf: function(doc) {
var date = new Date(doc.time);
var dateKey = (date.getMonth()+1)+"/"+date.getDate()+"/"+date.getFullYear();
return {'date': dateKey};
},
cond:query,
initial: {count:0},
reduce: function(obj, prev) {
prev.count++;
}
});
}
count_by_day({this:'is',the:'query'})
Another late answer, but still. So if you wanna do it in only one iteration and get the number of clicks grouped by date and topic you can use the following code:
db.coll.group(
{
$keyf : function(doc) {
return { "date" : doc.date.getDate()+"/"+doc.date.getMonth()+"/"+doc.date.getFullYear(),
"topic": doc.topic };
},
initial: {count:0},
reduce: function(obj, prev) { prev.count++; }
})
Also If you would like to optimize the query as suggested you can use an integer value for date (hint: use valueOf(), for the key date instead of the String, though for my examples the speed was the same.
Furthermore it's always wise to check the MongoDB docs regularly, because they keep adding new features all the time. For example with the new Aggregation framework, which will be released in the 2.2 version you can achieve the same results much easier http://docs.mongodb.org/manual/applications/aggregation/
If You want a Date oject returned directly
Then instead of applying the Date Aggregation Operators, instead apply "Date Math" to round the date object. This can often be desirable as all drivers represent a BSON Date in a form that is commonly used for Date manipulation for all languages where that is possible:
db.datetest.aggregate([
{ "$group": {
"_id": {
"$add": [
{ "$subtract": [
{ "$subtract": [ "$date", new Date(0) ] },
{ "$mod": [
{ "$subtract": [ "$date", new Date(0) ] },
1000 * 60 * 60 * 24
]}
]},
new Date(0)
]
},
"click": { "$sum": 1 }
}}
])
Or if as is implied in the question that the grouping interval required is "buckets" of 15 days, then simply apply that to the numeric value in $mod:
db.datetest.aggregate([
{ "$group": {
"_id": {
"$add": [
{ "$subtract": [
{ "$subtract": [ "$date", new Date(0) ] },
{ "$mod": [
{ "$subtract": [ "$date", new Date(0) ] },
1000 * 60 * 60 * 24 * 15
]}
]},
new Date(0)
]
},
"click": { "$sum": 1 }
}}
])
The basic math applied is that when you $subtract two Date objects the result returned will be the milliseconds of differnce numerically. So epoch is represented by Date(0) as the base for conversion in whatever language constructor you have.
With a numeric value, the "modulo" ( $mod ) is applied to round the date ( subtract the remainder from the division ) to the required interval. Being either:
1000 milliseconds x 60 seconds * 60 minutes * 24 hours = 1 day
Or
1000 milliseconds x 60 seconds * 60 minutes * 24 hours * 15 days = 15 days
So it's flexible to whatever interval you require.
By the same token from above an $add operation between a "numeric" value and a Date object will return a Date object equivalent to the millseconds value of both objects combined ( epoch is 0, therefore 0 plus difference is the converted date ).
Easily represented and reproducible in the following listing:
var now = new Date();
var bulk = db.datetest.initializeOrderedBulkOp();
for ( var x = 0; x < 60; x++ ) {
bulk.insert({ "date": new Date( now.valueOf() + ( 1000 * 60 * 60 * 24 * x ))});
}
bulk.execute();
And running the second example with 15 day intervals:
{ "_id" : ISODate("2016-04-14T00:00:00Z"), "click" : 12 }
{ "_id" : ISODate("2016-03-30T00:00:00Z"), "click" : 15 }
{ "_id" : ISODate("2016-03-15T00:00:00Z"), "click" : 15 }
{ "_id" : ISODate("2016-02-29T00:00:00Z"), "click" : 15 }
{ "_id" : ISODate("2016-02-14T00:00:00Z"), "click" : 3 }
Or similar distribution depending on the current date when the listing is run, and of course the 15 day intervals will be consistent since the epoch date.
Using the "Math" method is a bit easier to tune, especially if you want to adjust time periods for different timezones in aggregation output where you can similarly numerically adjust by adding/subtracting the numeric difference from UTC.
Of course, that is a good solution. Aside from that you can group dates by days as strings (as that answer propose) or you can get the beginning of dates by projecting date field (in aggregation) like that:
{'$project': {
'start_of_day': {'$subtract': [
'$date',
{'$add': [
{'$multiply': [{'$hour': '$date'}, 3600000]},
{'$multiply': [{'$minute': '$date'}, 60000]},
{'$multiply': [{'$second': '$date'}, 1000]},
{'$millisecond': '$date'}
]}
]},
}}
It gives you this:
{
"start_of_day" : ISODate("2015-12-03T00:00:00.000Z")
},
{
"start_of_day" : ISODate("2015-12-04T00:00:00.000Z")
}
It has some pluses: you can manipulate with your days in date type (not number or string), it allows you to use all of the date aggregation operators in following aggregation operations and gives you date type on the output.
I have the following schema in mongodb, where the timestamp is the timestamp at an hourly level
{
"day_chan1" : 54.464,
"day_chan2" : 44.141,
"day_chan3" : 44.89,
"gatewayId" : "443719005AA3",
"timestamp" : ISODate("2016-02-15T23:00:00.000Z"),
"total_curr_chan" : 5.408,
"total_day_chan" : 143.495,
"type" : 1
}
I want to be able to query the last timestamp for the day for the last 7 days and 30 days. In order to do this, I am thinking of doing something like
var d = new Date(); // today!
for(var i =1; i <= 7; i++) {
var n = i; // go back n days!
d.setDate(d.getDate() - n);
d.setHours(23,0,0);
var query = {
gatewayId: req.params.deviceId,
timestamp: { $lt: new Date(d) }
};
db
.find(query,function(resp) {
//return the data here
});
}
But this creates a problem of multiple callbacks and I want to know if there is an easier way of doing so using aggregates or some other method
Use the $hour operator within the $project operator to extract the hour part of the timestamp, then query with $match to filter documents that do not satisfy the given hour criteria:
var pipeline = [
{
"$project": {
"day_chan1": 1,
"day_chan2": 1,
"day_chan3": 1,
"gatewayId": 1,
"timestamp": 1,
"total_curr_chan": 1,
"total_day_chan": 1,
"type": 1,
"hour": { "$hour": "$timestamp" }
}
},
{ "$match": { "hour": 23 } }
];
collection.aggregate(pipeline, function(err, result) {
//return the data here
console.log(result);
});
For arbitrary last hour it must be a bit more complex:
db.collection.aggregate([
{$match:{
timestamp:{$type: "date"}}
// add date constraints here
},
{$project:{
_id:1,
date:{"y":{$year:"$timestamp"}, "d":{$dayOfYear:"$timestamp"}},
doc:"$$CURRENT"}
},
{$group:{
_id:"$date",
maxtime: {$max:"$doc.timestamp"},
doc:{$push:"$doc"}}
},
{$unwind:"$doc"},
{$project:{
latest: {$cmp: ["$maxtime", "$doc.timestamp"]},
doc:"$doc"}
},
{$match:{"latest":0}}
])
With map-reduce it should be simpler, but may be slower.
I have a MongoDB whom store the date objects in UTC. Well, I want to perform aggregation by year,month day in a different timezone (CET).
doing this, works fine for UTC:
BasicDBObject group_id = new BasicDBObject("_id", new BasicDBObject("year", new BasicDBObject("$year", "$tDate")).
append("month", new BasicDBObject("$month", "$tDate")).
append("day", new BasicDBObject("$dayOfMonth", "$tDate")).
append("customer", "$customer"));
BasicDBObject groupFields = group_id.
append("eventCnt", new BasicDBObject("$sum", "$eventCnt"));
BasicDBObject group = new BasicDBObject("$group", groupFields);
or, if you use the command line (not tested, I only tested the java version):
{
$group: {
_id: {
"year": {
"$year", "$tDate"
},
"month": {
"$month", "$tDate"
},
"day": {
"$dayOfMonth", "$tDate"
},
"customer": "$customer"
},
"eventCount": {
"$sum": "$eventCount"
}
}
}
How do I convert these dates into CET inside the aggregation framework?
For example '2013-09-16 23:45:00 UTC' is '2013-09-17 00:45:00 CET', this is a different day.
I'm not an expert on CET and its relation to UTC, but the following code (for the shell) should do a proper conversion (adding an hour) to a MongoDB date type:
db.dates.aggregate(
{$project: {"tDate":{$add: ["$tDate", 60*60*1000]}, "eventCount":1, "customer":1}}
)
If you run that project command before the rest of your pipeline, the results should be in CET.
You can provide the timezone to the date operators starting in 3.6.
Replace the timezone with your timezone.
{
"$group":{
"_id":{
"year":{"$year":{"date":"$tDate","timezone":"America/Chicago"}},
"month":{"$month":{"date":"$tDate","timezone":"America/Chicago"}},
"dayOfMonth":{"$dayOfMonth":{"date":"$tDate","timezone":"America/Chicago"}}
},
"count":{"$sum":1}
}
}
After searching for hours, this is the solution that worked for me. It is also very simple. Just convert the timezone by subtracting the timezone offset in milliseconds.
25200000 = 7 hour offset // 420 min * 60 sec * 1000 mili
$group: {
_id = {
year: { $year : [{ $subtract: [ "$timestamp", 25200000 ]}] },
month: { $month : [{ $subtract: [ "$timestamp", 25200000 ]}] },
day: { $dayOfMonth : [{ $subtract: [ "$timestamp", 25200000 ]}] }
},
count = {
$sum : 1
}
};
Use for example moment.js to dertmine the current timezone offset for CET but this way you get the summer&winter offsets
var offsetCETmillisec = moment.tz.zone('Europe/Berlin').offset(moment())* 60 * 1000;
$group: {
_id: {
'year': {'$year': [{ $subtract: [ '$createdAt', offsetCETmillisec ]}] },
'month': {'$month': [{ $subtract: [ '$createdAt', offsetCETmillisec ]}] },
'day': {'$dayOfMonth': [{ $subtract: [ '$createdAt', offsetCETmillisec ]}] }
},
count: {$sum: 1}
}
}
MongoDB's documentation suggests that you save the timezone offset alongside the timestamp:
var now = new Date();
db.data.save( { date: now,
offset: now.getTimezoneOffset() } );
This is of course not the ideal solution – but one that works, until we have in MongoDb's aggregation pipeline a proper $utcOffset function.
The solution with timezone is a good one, but in version 3.6 you can also format the output using timezone, so, you get the result ready for use:
{
"$project":{
"year_month_day": {"$dateToString": { "format": "%Y-%m-%d", "date": "$tDate", "timezone": "America/Chicago"}}
},
"$group":{
"_id": "$year_month_day",
"count":{"$sum":1}
}
}
Make sure that your "$match" also considers timezone, or else you will get wrong results.
Mongo stores the dates in UTC,
so this is the procedure to get them in other zone
check that mongo saves the dates in UTC, insert some records etc.
get timezone offset with moment-timezone.js eg moment().tz('Europe/Zagreb').utcOffset() functions, for your
specified timezone
Prepare $gte and $lte for $match stage (eg user input for dates 1.1.2019 - 13.1.2019.):
If offset is positive subtract() those seconds in $match stage; If offset is negative add() those seconds in $match stage
Then normalize the dates (because $match stage will return them in UTC) to your zone like this:
-if timezone offset is positive add() those seconds in $project stage;
-if timezone offset is negative subtract() those seconds in $project stage.
$group goes last, this is important (because we want to group normalized results, and not $match-ed)
Basically it is this: shift input(s) to $match(UTC), and then normalize to your timezone.
<?php
date_default_timezone_set('Asia/Karachi');
$date=getdate(date("U"));
$day = $date['mday'];
$month =$date['mon'];
$year = $date['year'];
$currentDate = $year.'-'.$month.'-'.$day;
?>