I am trying this aggregation last 4 months records each month OZONE average value but average value is null how to get the average value
db.Air_pollution.aggregate([
{$match:{CREATE_DATE:{$lte:new Date(),$gte:new Date(new Date().setDate(new
Date().getDate()-120))}}},
{$group:{_id:{month:{$month:"$CREATE_DATE"},year:
{$year:"$CREATE_DATE"}},avgofozone:{$avg:"$OZONE"}}},
{$sort:{"year":-1}},{$project:
{year:'$_id.year',month:'$_id.month',_id:0,avgofozone:1}}
])
output:
{ "avgofozone" : null, "year" : 2018, "month" : 2 }
{ "avgofozone" : null, "year" : 2018, "month" : 3 }
{ "avgofozone" : null, "year" : 2018, "month" : 1 }
It's not working because the OZONE field is a string, and you can't compute $avg on a string. Plus, it's not a valid number: "8:84" should be 8.84
from mongodb documentation:
$avg
Returns the average value of the numeric values that result from
applying a specified expression to each document in a group of
documents that share the same group by key. $avg ignores non-numeric
values.
Otherwise the aggregation query is correct, here is a link showing it: mongo playground.net/p/VaL-Nn8e21E
Related
With the following data structure, using mongoDB's (v3.4) aggregation framework how do you group information every 15 days?
{
"_id" : ObjectId("5cb10a201e20af7503305fea"),
"user" : ObjectId("5b21240c4e71161fdd40b27c"),
"version" : NumberLong(2),
"value" : 42,
"itemRef" : ObjectId("5cb10a201e20af7503305fe9"),
"status" : "ACCEPTED",
"date" : ISODate("2019-04-13T11:00:00.466Z")
}
the required output would be:
[date: 2019/01/01, totalValue:15],
[date: 2019/01/16, totalValue:5],
[date: 2019/02/01, totalValue:25],
[date: 2019/02/16, totalValue:30]
The way I found to resolve this problem with mongoDB 3.4 was using $cond + $dayOfMonth to define in which part of the month this date is.
db.contract.aggregate(
[
{$match:{...queryGoesHere...}},
{$project:
{dateText:
{$cond:
[
{$lte:[{$dayOfMonth:$date},15]},
['$dateToString': ['format': '%Y-%m-01', 'date': '$date']],
['$dateToString': ['format': '%Y-%m-16', 'date': '$date']]
]
}
value:'$value'
}
},
{$group:
{
_id:'$dateText',
total:{'$sum':1}
}
}
]
The solution is in the projection of the "dateText", it first uses the $cond to determine if the date is in the first or second part of the month. It determines this using the '$dayOfMonth' which returs the day in the month. If it is less or equal to 15, it uses the '$dateToString' to format the date by year-month-01 else it formats it to year-month-16.
Hope this can help someone in the future.
I have a lot of devices non-periodically inserting data into mongo.
I need to get statistics of this data (value by day/month/year). Currently i am doing this by adding a field where I parse the date to day month and year using $month, $year, $dayOfMonth. Then grouping them by these values. The problem is when I get no (or only one) data a day. Then I cant get actual value in this day because I need 2 values to subtract.
Is there a way to get the closest document by day to this group? in one query?
Lets say I have data:
{id : 1, ts : "2017-12-15T10:00:00.000Z", value : 10}
{id : 2, ts : "2017-12-15T17:00:00.000Z", value : 10}
{id : 2, ts : "2017-12-14T12:00:00.000Z", value : 6}
{id : 1, ts : "2017-12-14T15:00:00.000Z", value : 10}
{id : 1, ts : "2017-12-14T10:00:00.000Z", value : 10}
{id : 2, ts : "2017-12-14T09:00:00.000Z", value : 3}
Explanation of problem:
The value is actual read from the meter, for example lets say consumed energy. If device sonsumes 4W/min after 1 min it will be 4 after 2 minutes it will be 8. So the delta between 1. and 2. minute will be 4 . If i have record from 2017-12-14T23:58:00.000Z lets say 10W 23:59 it will be 14W so dValue should be 4 and 00:00 the next day i am not able to calculate the dValue because this is the first and only record in this group
If I group this data by day I can calculate the value difference only in 2017-12-14.
For now I am using this query:
{
$addFields : {
month : {$month : "$ts"},
year : {$year : "$ts"},
day : {$dayOfMonth : "$ts"}
}
},
{
$group : {
_id : {
year : "$year",
month : "$month",
day : "$day",
id : "$id"
},
first : {$min : "$$ROOT"},
last : {$max : "$$ROOT"},
}
},
{
$addFields : {
dValue: {$subtract : [last.value, first.value]} //delta value
}
},
This query works but only if there is more than one document in a day. If there is only one document i cant get accurate data. I want to do this in one query, because i have a lot of these devices and the number is going to only increase and if i have to do a query for every device i get insane number of queries to the database. Is there a way how to solve this ?
This question already has answers here:
Group result by 15 minutes time interval in MongoDb
(7 answers)
Closed 5 years ago.
I have a collection of objects in my mongoDB containing datetimes among other values.
How would I go about querying the objects by datetime, where the timestamp is set to be at 9 o'clock?
So if I have the following collection...
id : 1, date : ISODate("2017-07-16T09:00:00.000+0000")
id : 2, date : ISODate("2017-01-17T07:00:00.000+0000")
id : 3, date : ISODate("2017-07-27T09:00:00.000+0000")
id : 4, date : ISODate("2017-03-20T09:00:00.000+0000")
id : 5, date : ISODate("2017-03-07T10:00:00.000+0000")
id : 6, date : ISODate("2017-07-04T11:00:00.000+0000")
The return value should be...
id : 1, date : ISODate("2017-07-16T09:00:00.000+0000")
id : 3, date : ISODate("2017-07-27T09:00:00.000+0000")
id : 4, date : ISODate("2017-03-20T09:00:00.000+0000")
I'm fairly new to MongoDB and not very experienced with Js so please try and keep it as simple as you can. To that note Neil Lunn marked this question as a duplicate of
This Question, which I feel is partially correct, but it's also more complex than I need.
I don't need grouping or anything of that nature, I just want a query that tells me which documents exist containing this timestamp.
You could use an aggregate pipeline to convert the date to its timepart and then match on that converted value. For example:
db.collection.aggregate([
{
$project: {
timePart: {$dateToString: { format: "%H:%M:%S:%L", date: "$date"}},
date: 1
}
},
{
$match: {
timePart: '09:00:00:000'
}
},
{
$project: {
date: 1
}
}
])
You can think of this as a pipeline; the output from the first $project step becomes the input to the $match step. The $project step outputs - for every document in the underlying collection - a document containing the _id, the date and a new attribute named timePart which has been populated with the time part from the date attribute. The $match step then matches these documents against your filter criteria (in your example this is 09:00:00:000 i.e. 9am) and then the documents which are matched are then forwarded to the next step which uses the $project operator again to discard the timePart attribute since, I assume, that is only relevant for saerching and should not be included in the end result.
Breaking it down, the output of the first step looks like this:
{
"_id" : 1,
date : ISODate("2017-07-16T09:00:00.000+0000"),
timePart: "09:00:00.000"
},
{
"_id" : 2,
date : ISODate("2017-01-17T07:00:00.000+0000"),
timePart: "07:00:00.000"
},
...
The second step excludes the document with id: 2 because its timePart does not match 09:00:00.000 and then forwards the document with id: 1 to the third stage which then selects - from those documents forwarded by step 2 - the fields _id and date thereby giving you:
{
"_id" : 1,
date : ISODate("2017-07-16T09:00:00.000+0000")
},
{
"_id" : 3,
date : ISODate("2017-07-27T09:00:00.000+0000")
},
{
"_id" : 4,
date : ISODate("2017-03-20T09:00:00.000+0000")
}
Note: this approach must transform the date attribute of every document before applying the match stage, if that's worryingly inefficient for you then you might want to reconsider how you are persisting this data.
is it possible to query only the first (or last or any single?) day of the month of a mongo date field.
i use the $date aggregation operators regularly but within a $group clause.
basically i have field that is already aggregated (averaged) for each day of the month. i want to select only one of these days (with the value as a representative of the entire month.)
following is a sample of a record set from jan 1, 2014 to feb 1, 2015 with price as the daily price and 28day_avg as the trailing monthly average for 28 days.
{ "date" : ISODate("2014-01-01T00:00:00Z"), "_id" : ObjectId("533b3697574e2fd08f431cff"), "price": 59.23, "28day_avg": 54.21}
{ "date" : ISODate("2014-01-02T00:00:00Z"), "_id" : ObjectId("533b3697574e2fd08f431cff"), "price": 58.75, "28day_avg": 54.15}
...
{ "date" : ISODate("2015-02-01T00:00:00Z"), "_id" : ObjectId("533b3697574e2fd08f431cff"), "price": 123.50, "28day_avg": 122.25}
method 1.
im currently running an aggregation using $month data (and summing the price) but one issue is im seeking to retrieve the underlying date value ISODate("2015-02-01T00:00:00Z") versus the 0,1,2 value that comes with several of the date aggregations (that loop at the first of the week, month, year). mod(28) on a date?
method 2
i'd like to simply pluck out a single record of the 28day_avg as representative of the period. the 1st of the month would be adequate
the desired output is...
_id: ISODate("2015-02-01T00:00:00Z"), value: 122.25,
_id: ISODate("2015-01-01T00:00:00Z"), value: 120.78,
_id: ISODate("2014-12-01T00:00:00Z"), value: 118.71,
...
_id: ISODate("2014-01-01T00:00:00Z"), value: 53.21,
of course, the value will vary from method 1 to method 2 but that is fine. one is 28 days trailing while the other will account for 28, 30, 31 day months...dont care about that so much.
A non-agg is ok but also doesnt work. aka {"date": { "$mod": [ 28, 0 ]} }
To pick the first of the month for each month (method 2), use the following aggregation:
db.test.aggregate([
{ "$project" : { "_id" : "$date", "day" : { "$dayOfMonth" : "$date" }, "28day_avg" : 1 } },
{ "$match" : { "day" : 1 } }
])
You can't use an index for the match, so this is not efficient. I'd suggest adding another field to each document that holds the $dayOfMonth value, so you can index it and do a simple find:
{
"date" : ISODate("2014-01-01T00:00:00Z"),
"price" : 59.23,
"28day_avg" : 54.21,
"dayOfMonth" : 1
}
db.test.ensureIndex({ "dayOfMonth" : 1 })
db.test.find({ "dayOfMonth" : 1 }, { "_id" : 0, "date" : 1, "28day_avg" : 1 })
Is there a method of grouping a Unix epoch time by day, equiv to $dayOfYear
or a process of aggregating floats, ints (into quartiles, hundreds, thousands, %)
try to avoid map reduce but an example of it would be awesome.
You can almost but not quite use Unix time seconds in aggregation pipeline by utilizing the $mod and $divide operators.
The math is Unix time seconds / 86400 to convert seconds into days since Epoch. Then modula that result by 365.25 for the day of the year (leaps every 4).
So the full aggregation for $dayOfYear using seconds is almost as simple as
db.MyCollection.aggregate( {$project : {"day" : {$mod : [ {$divide : ["$unix_seconds", 86400] } , 365.25] } } }, { $group : { _id : "$day" , num : { $sum : 1 } } } , {$sort : {_id : 1}} )
The above adds sorting for sequential day of year.
The problem is that the $mod operator returns both the whole number and remainder. and there is no way of rounding or truncating the remainder. Therefore the results are grouped by whole and remainder.
{
"_id" : 235.1864887063916,
"num" : 1
},
{
"_id" : 235.24300889818738,
"num" : 1
},
{
"_id" : 235.60299520864623,
"num" : 3
},
{
"_id" : 235.66453935674085,
"num" : 1
},
{
"_id" : 235.79900382758004,
"num" : 1
},
{
"_id" : 235.80265845312474,
"num" : 1
},
.. when clearly we want only the whole number
{
"_id" : 235,
"num" : 8
},
What would be nice is a $trunc or modula returning only the whole ($modw), and mod returning only remainder ($modr) operators in mongo.
JavaScript has the Date object which would be available to any server side JavaScript processing for MapReduce functions.
You seem to be aware of the $dayOfYear operator in the aggregation pipeline. There are other operators there for processing dates.
Unless your needs are very specific you should be using the aggregation pipeline. It is very flexible and in most cases will be considerably faster than the equivalent actions run under mapReduce.