I have switched sql to mongodb recently and I ran into an issue.
Search schema:
new mongoose.Schema({
content: {
type: String,
required: true
},
created_at: {
type: Date,
default: Date.now
})
Some docs:
_id:5e077923f08564374cea9f55
content:"ds"
created_at:2019-12-28T15:47:47.318+00:00
_id:5e077924f08564374cea9f56
content:"ds"
created_at:2019-12-28T15:47:48.545+00:00
_id:5e07792bf08564374cea9f57
content:"d"
created_at:2019-12-28T15:47:55.585+00:00
How can I count search IN A DAY / WEEK / MONTH.
I have an idea, try to convert date to int. But I don't know how to do it.
Thank you very much.
You can use $year, $month to group your date field by year, month and then calculate count for this group. One sample query would look like this -
db.col.aggregate([
{
$group: {
"_id": {
"year" : {
$year : "$createdDate"
},
"month" : {
$month : "$createdDate"
}
},
"count": {
$sum: 1
}
}
}
])
How can I count documents by specific date range
You can specify a range of dates, start and end (and this range can include any number of days: one day, 7 days, 30 days, etc.) and get a count for that.
var start = ISODate("2019-12-25T09:00:00.000+00:00")
var end = ISODate("2019-12-30T17:00:00.000+00:00")
db.dates.aggregate( [
{
$match: {
$and: [ { created_at: { $gte: start } },
{ created_at: { $lte: end } }
]
}
},
{
$group: {
_id: null,
count: { $sum: 1 }
}
}
] )
The query can also be written as a single $group stage:
{
$group: {
_id: null,
count: {
$sum: {
$cond: [
{ $and: [
{ $gte: [ "$created_at", start ] },
{ $lte: [ "$created_at", end ] }
] },
1,
0
]
}
}
}
}
Related
{
metadata:{
dat:jkjcsvbdskjcbdskjcbdac,
meterId:kahcvajc
}
activeEnergy:1111,
actualtime:1689827191000
}
The document is something like that, I am facing problem only with activeEnergy so I want to focus on that only. Below I have written the code, in the first group object I have divided by year, month, day but in actual code it is dynamic, if the payload from frontend I receive is month and week then I group accordingly, but if I receive day in payload then I group by hour, I calculate the max and min energy of that hour and then sum for all hour, as activeEnergy is continuous, but the problem is that I am not getting data at every second so it is possible that I get the first data at 10:25am and last data at 10:45am therefore by taking max and min I only calculate for that 20 min and miss on all the remaining time data, ideally what I should do is max of this hour by max of previous hour. Similarly for the day the first document is at 12:59 AM and last Data at 11 pm, IF I calculate the max and min of the day I will still miss 2 hour data approx, so I have to find a way to find difference of max of today and max of previous day.
That becomes a problem because data is not grouped like that by me and I don't know any other way I can group data, how to solve the problem?
db.ts_events.aggregate([
{
$project: {
"y":{"$year": {$toDate: "$actualtime"}},
"m":{"$month": {$toDate: "$actualtime"}},
"d":{"$dayOfMonth": {$toDate: "$actualtime"}},
"h":{"$hour": {$toDate: "$actualtime"}},
"activeEnergy": 1,
"metadata.meterId": 1,
"activePower": 1,
"actualtime": 1,
"powerFactor": 1,
"metadata.dat": 1
}
},
{
$match: {
"metadata.dat": "62f0f3459731692a5eab5ad6/south0tpbit/tamilnadu5dvs8w/chennaidzc2yd/kknagarj4ffzo",
"actualtime": {
$gte: 1656613800000, $lte: 1659292199999,
},
// "metadata.device":"ObjectId(62f0f9b5f757672222282d9)" how to check using object id?,
"metadata.meterId": "911615402222257_2",
}
},
{
$group: {
_id: {
date: {
year: "$y",
month: "$m",
day: "$d",
// hour: "$h",
},
meter: "$metadata.meterId",
},
maxValue: {
$max: "$activeEnergy"
},
minValue: {
$min: "$activeEnergy"
},
averageActivePowerOfDay: { $avg: "$activePower" },
averagePowerFactorOfDay: { $avg: "$powerFactor" },
}
},
{
$addFields: {
differnce: {
$subtract: [
"$maxValue",
"$minValue"
]
},
}
},
//
//
//
{
$group: {
_id: null, res: {
$push: '$$ROOT'
}, differnceSum: {
$sum: '$differnce'
},
averageActivePowerOverThePeriod: {
$avg: "$averageActivePowerOfDay"
},
averagePowerFactorOverThePeriod: {
$avg: "$averagePowerFactorOfDay"
}
}
}
])
You can try with $setWindowFields
db.collection.aggregate([
{ $set: { actualTime: { $toDate: "$actualtime" } } },
{
$setWindowFields: {
partitionBy: "$metadata.dat",
sortBy: { actualTime: 1 },
output: {
max_today: {
$max: "$activeEnergy",
window: { range: [-1, 1], unit: "day" }
},
max_yesterday: {
$max: "$activeEnergy",
window: { range: [-2, -1], unit: "day" }
},
day: {
$last: "$actualTime",
window: { range: [-1, 1], unit: "day" }
},
}
}
},
{
$group: {
_id: { metadata: "$metadata", day: "$day" },
values: {
$addToSet: {
max_today: "$max_today",
max_yesterday: "$max_yesterday"
}
}
}
},
{ $replaceWith: { $mergeObjects: ["$_id", { $first: "$values" }] } }
])
I have data of various timestamps and I want to create an aggregate pipeline to get sum of a column having yesterdays date. And I don't want to hardcode current date to get yesterday date.
Please suggest how to do it as I am new to Mongodb
Edit :
collection name - consumption_data
documents -
1. id :101, timestamp : 2022-09-10T22:00:00.000+00:00, consumption: 199
2. id :106, timestamp : 2022-09-10T07:00:00.000+00:00, consumption: 201
3. id :108, timestamp : 2022-09-11T12:00:00.000+00:00, consumption: 77
4. id :109, timestamp : 2022-09-11T08:00:00.000+00:00, consumption: 773
If today is 2022-09-11 the I want consumption of yesterday(2022-09-10) without hardcoding the dates
Try this one:
db.consumption_data.aggregate([
{
$match: {
$expr: {
$gt: ["$timestamp", {
$dateSubtract: {
startDate: "$$NOW",
unit: "day",
amount: 1
}
}]
}
}
},
{ $group: { _id: null, consumption: { $sum: "$consumption" } } }
])
Consider the use of $dateTrunc, (i.e. { $dateTrunc: { date: "$$NOW", unit: "day" } }) otherwise it will go back exactly 24 hours from the current time
db.consumption_data.aggregate([
{
$match: {
$expr: {
$and: [
{
$gte: ["$timestamp",
{
$dateSubtract: {
startDate: { $dateTrunc: { date: "$$NOW", unit: "day" } },
unit: "day",
amount: 1
}
}
]
},
{
$lt: ["$timestamp",
{ $dateTrunc: { date: "$$NOW", unit: "day" } }
]
}
]
}
}
},
{ $group: { _id: null, consumption: { $sum: "$consumption" } } }
])
Here is a working solution:
db.totalConsumption_data.aggregate([
{
'$project': {
'month': {
'$month': '$timestamp'
},
'year': {
'$year': '$timestamp'
},
'day': {
'$dayOfMonth': '$timestamp'
},
'timestamp': 1,
'consumption': 1
}
}, {
'$match': {
'month': new Date().getMonth() +1,
'day': new Date().getDate()-1,
'year': new Date().getFullYear()
}
}, {
'$group': {
'_id': null,
'total': {
'$sum': '$consumption'
}
}
}
]
I'm trying to sum the 'emsion_co2' values by day. in order to know in what days the value is higher than 420, but my query is not working.
this is the database
db.test_sensores.insert([{"timestamp":"2020-07-01T00:00:00Z","dia":01,"sensor_id":1,"location_id":1,"medidas":[{"tipo_medida":"Temperatura","valor":16.61,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":83.74,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:00:00Z","dia":01,"sensor_id":2,"location_id":1,"medidas":[{"tipo_medida":"Emision_CO2","valor":1.572,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00188,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-01T00:15:00Z","dia":01,"sensor_id":1,"location_id":1,"medidas":[{"tipo_medida":"Temperatura","valor":15.75,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":83.08,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:15:00Z","dia":01,"sensor_id":2,"location_id":1,"medidas":[{"tipo_medida":"Emision_CO2","valor":1.626,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00146,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-10T00:30:00Z","dia":10,"sensor_id":1,"location_id":1,"medidas":[{"tipo_medida":"Temperatura","valor":14.08,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":90.05,"unidad":"%"}]},
{"timestamp":"2020-07-10T00:30:00Z","dia":10,"sensor_id":2,"location_id":1,"medidas":[{"tipo_medida":"Emision_CO2","valor":1.614,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00173,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-01T00:45:00Z","dia":01,"sensor_id":1,"location_id":1,"medidas":[{"tipo_medida":"Temperatura","valor":10.62,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":92.27,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:45:00Z","dia":01,"sensor_id":2,"location_id":1,"medidas":[{"tipo_medida":"Emision_CO2","valor":1.59,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00196,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-01T00:00:00Z","dia":01,"sensor_id":1,"location_id":2,"medidas":[{"tipo_medida":"Temperatura","valor":22.08,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":34.92,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:00:00Z","dia":01,"sensor_id":2,"location_id":2,"medidas":[{"tipo_medida":"Emision_CO2","valor":2.055,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00269,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-01T00:15:00Z","dia":01,"sensor_id":1,"location_id":2,"medidas":[{"tipo_medida":"Temperatura","valor":21.12,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":37.7,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:15:00Z","dia":01,"sensor_id":2,"location_id":2,"medidas":[{"tipo_medida":"Emision_CO2","valor":2.102,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00272,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-10T00:30:00Z","dia":10,"sensor_id":1,"location_id":2,"medidas":[{"tipo_medida":"Temperatura","valor":24.26,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":52.52,"unidad":"%"}]},
{"timestamp":"2020-07-10T00:30:00Z","dia":10,"sensor_id":2,"location_id":2,"medidas":[{"tipo_medida":"Emision_CO2","valor":2.023,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00255,"unidad":"kWh/m2"}]},
{"timestamp":"2020-07-01T00:45:00Z","dia":01,"sensor_id":1,"location_id":2,"medidas":[{"tipo_medida":"Temperatura","valor":24.51,"unidad":"ºC"},{"tipo_medida":"Humedad_relativa","valor":34.1,"unidad":"%"}]},
{"timestamp":"2020-07-01T00:45:00Z","dia":01,"sensor_id":2,"location_id":2,"medidas":[{"tipo_medida":"Emision_CO2","valor":1.957,"unidad":"gCO2/m2"},{"tipo_medida":"Consumo_electrico","valor":0.00215,"unidad":"kWh/m2"}]}])
And the next query is what I was trying, but the result is wrong because there is not showing the sum, just appear in 0 the values.
db.test_sensores.aggregate([{$match:{'medidas.tipo_medida':'Emision_CO2'}},{$group:{_id:'$dia', acumulado_emisiones_CO2:{$sum:'$medidas.0.valor'}}}])
In my query is missing how to find the days where emision_co2 value > 420 too, I was thinking manually but if there is an automated way, please let me know. Thank you
Unsure why you attempt to group by dia,
$match
$set - Set timestamp to Date object, medidas_Emision_CO2_valor for sum all the values in medidas array by filtering ($filter) the document with tipo_medida is "Emision_CO2" via $reduce.
$group - Group by date, month, and year of timestamp and sum the medidas_Emision_CO2_valor values as acumulado_emisiones_CO2.
$match - Filter the documents with acumulado_emisiones_CO2 is greater than ($gt) 420.
db.test_sensores.aggregate([
{
$match: {
"medidas.tipo_medida": "Emision_CO2"
}
},
{
$set: {
timestamp: {
$toDate: "$timestamp"
},
medidas_Emision_CO2_valor: {
$reduce: {
input: {
$filter: {
input: "$medidas",
cond: {
$eq: [
"$$this.tipo_medida",
"Emision_CO2"
]
}
}
},
initialValue: 0,
in: {
$sum: [
"$$value",
"$$this.valor"
]
}
}
}
}
},
{
$group: {
_id: {
day: {
"$dayOfMonth": "$timestamp"
},
month: {
"$month": "$timestamp"
},
year: {
$year: "$timestamp"
}
},
acumulado_emisiones_CO2: {
$sum: "$medidas_Emision_CO2_valor"
}
}
},
{
$match: {
"acumulado_emisiones_CO2_valor": {
$gt: 420
}
}
}
])
Sample Mongo Playground
my collection in db is
plannedEndDate:{"2020-03-10T11:22:33.677+00:00"}
in controller var tdate= new Date(); [tdate is in the format 2020-03-10T14:28:22.687Z].
now,I need to check plannedEndDate is less than tdate in mongoose.
i tried,
plannedEndCmp: {
$cond: [{ $lt: ["$plannedEndDate", tdate] }, 1, 0]
}
but it is not returning true.
so should i need to trim the timestamp to compare dates in mangodb?or should i need to convert date into common format?
Not clear what you actually like to do but below command find the document or show indication:
db.collection.find(
{ plannedEndDate: { $lt: tdate } }
)
db.collection.aggregate([
{ $set: { plannedEndCmp: { $cond: [{ $lt: ["$plannedEndDate", tdate] }, 1, 0] } } },
{ $set: { plannedEndCmp_bool: { $lt: ["$plannedEndDate", tdate] } } }
])
In Aggregate group:
`
factualEndDate: {
"$dateToString": { "format": "%Y-%m-%d", "date": "$factualEndDate" }
},
In Aggregate project:
DelayedComplete: { $cond: [
{
$and: [
{ $cond: [{ $lte: [{ $max: "$data.factualEndDate" }, today] }, 1, 0] },
]
},
1,
0
]
},
`
I have the following documents stored in a collection:
{
"REQUESTTIMESTAMP" : "26-JUN-19 01.34.10.095000000 AM",
"UNHANDLED_INTENT" : 0,
"USERID" : "John",
"START_OF_INTENT_SKILL_CONVERSATION" : 0,
"PROPERTYCODE" : ""
}
I want to group this by the hour(which we will get from 'REQUESTTIMESTAMP')
Earlier, I had this document stored in the collection in a different way, where I had a separate field for hours, and used that hours field to group:
Previous aggregation query :
collection.aggregate([
{'$match': query}, {
'$group': {
"_id": {
"hour": "$hour",
"sessionId": "$sessionId"
}
}
}, {
"$group": {
"_id": "$_id.hour",
"count": {
"$sum": 1
}
}
}
])
Previous collection structure:
{
"timestamp" : "1581533210921",
"date" : "12-02-2020",
"hour" : "13",
"month" : "02",
"time" : "13:46:50",
"weekDay" : "Wednesday",
"__v" : 0
}
How can I do the above same Previous aggregation query with the new document structure (After extracting hours from 'REQUESTTIMESTAMP' field?)
You should convert your timestamp to Date object then take hour from your date object.
db.collection.aggregate([{
'$match': query
}, {
$project: {
date: {
$dateFromString: {
dateString: '$REQUESTTIMESTAMP',
format: "%m-%d-%Y" //This should be your date format
}
}
}
}, {
$group: {
_id: {
hour: {
$hour: "$date"
}
}
}
}])
Problem is months names are not supported by MongoDB. Either you write a lot of code or you use libraries like moments.js. First update your REQUESTTIMESTAMP to proper Date object, then you can group it.
db.collection.find().forEach(function (doc) {
var d = moment(doc.REQUESTTIMESTAMP, "DD-MMM-YY hh.mm.ss.SSS a");
db.collection.updateOne(
{ _id: doc._id },
{ $set: { date: d.toDate() } }
);
})
db.collection.aggregate([
{
$group: {
_id: { $hour: "$date" },
count: { $sum: 1 }
}
}
])
In case if you're not able to update DB with actual date field & still wanted to proceed with existing format, try this query it will add hour field extracted from given string field REQUESTTIMESTAMP :
Query :
db.collection.aggregate([
{
$addFields: {
hour: {
$let: {
/** split string into three parts date + hours + AM/PM */
vars: { hour: { $slice: [{ $split: ["$REQUESTTIMESTAMP", " "] }, 1, 2] } },
in: {
$cond: [{ $in: ["AM", "$$hour"] }, // Check AM exists in array
{ $toInt: { $substr: [{ $arrayElemAt: ["$$hour", 0] }, 0, 2] } }, // If yes then return int of first 2 letters of first element in hour array
{ $add: [{ $toInt: { $substr: [{ $arrayElemAt: ["$$hour", 0] }, 0, 2] } }, 12] } ] // If PM add 12 to int of first 2 letters of first element in hour array
}
}
}
}
}
])
Test : MongoDB-Playground