MongoDB Match DATE between FAILS - mongodb

I have a match like:
{
$and: [
{ $nor: [ { Meetings: { $exists: false } }, { Meetings: { $size: 0 } }, { Meetings: { $eq: null } } ] },
{ 'Meetings.MeetingDate': { $gte: ISODate("2020-12-23T00:00:01.000Z") } },
{ 'Meetings.MeetingDate': { $lte: ISODate("2020-12-23T23:59:59.999Z") } }
]
}
and on Mongo I have meetings from 2020-01-01 to 2020-12-31.
If I want to get only the 23rd ones, this match brings them but also from higer date like 25, 26, 30, etc...
What is the correct way to match date BETWEEN to get a specific date? (could be one day or a range...)
Here there is a Mongo Playground with a small example, but here works fine, I get all from the 29th.
I guess my problem is in my Aggregation. On the example I added MeetingDate on the root and in real life its a child array, maybe this is the problem.
db.getCollection("ClientProject").aggregate(
[
{
"$match" : {
"$and" : [
{
"$nor" : [
{
"Meetings" : {
"$exists" : false
}
},
{
"Meetings" : {
"$size" : 0.0
}
},
{
"Meetings" : {
"$eq" : null
}
}
]
},
{
"Meetings.MeetingDate" : {
"$gte" : ISODate("2020-12-30T00:00:01.000+0000")
}
},
{
"Meetings.MeetingDate" : {
"$lte" : ISODate("2020-12-31T23:59:59.999+0000")
}
}
]
}
},
{
"$project" : {
"ProjectName" : 1.0,
"ClientName" : 1.0,
"ClientResponsableName" : "$CreatedByName",
"ProjectType" : 1.0,
"ProjectSKU" : 1.0,
"Meetings" : 1.0
}
},
{
"$unwind" : {
"path" : "$Meetings",
"preserveNullAndEmptyArrays" : false
}
},
{
"$unwind" : {
"path" : "$Meetings.Invites",
"preserveNullAndEmptyArrays" : false
}
},
{
"$addFields" : {
"Meetings.Invites.MeetingDate" : "$Meetings.MeetingDate",
"Meetings.Invites.MeetingStartTime" : "$Meetings.StartTime",
"Meetings.Invites.MeetingEndTime" : "$Meetings.EndTime",
"Meetings.Invites.MeetingStatus" : "$Meetings.MeetingStatus",
"Meetings.Invites.ProjectId" : {
"$toString" : "$_id"
},
"Meetings.Invites.ProjectType" : "$ProjectType",
"Meetings.Invites.ProjectSKU" : "$ProjectSKU",
"Meetings.Invites.ProjectName" : "$ProjectName",
"Meetings.Invites.ClientId" : "$ClientId",
"Meetings.Invites.ClientName" : "$ClientName",
"Meetings.Invites.ClientResponsableName" : "$ClientResponsableName"
}
},
{
"$replaceRoot" : {
"newRoot" : "$Meetings.Invites"
}
},
{
"$sort" : {
"MeetingDate" : 1.0,
"MeetingStartTime" : 1.0,
"InviteStatus" : 1.0
}
}
],
{
"allowDiskUse" : false
}
);
cheers

The problem here is not related with the match on Date, it´s working. The problem is that each record has an array of meeting and each one has different dates, so even matching the correct date, the rest of the aggregation, Unwinds, etc... will use the the full record with all meetings, thats why I get with diferent dates.
Here I added an extra Match after $ReplaceRoot and worked... could use $filter too in some part of the aggregation..
cheers

Related

Creating measures in a mongodb aggregation pipeline

I have a report that has been developed in PowerBI. It runs over a collection of jobs, and for a given month and year counts the number of jobs that were created, due or completed in that month using measures.
I am attempting to reproduce this report using a mongoDB aggregation pipeline. At first, I thought I could just use the $group stage to do this, but quickly realised that grouping by a specific date would exclude jobs.
Some sample documents are below (most fields excluded as they are not relevant):
{
"_id": <UUID>,
"createdOn": ISODate("2022-07-01T00:00"),
"dueOn": ISODate("2022-08-01T00:00"),
"completedOn": ISODate("2022-07-29T00:00")
},
{
"_id": <UUID>,
"createdOn": ISODate("2022-06-01T00:00"),
"dueOn": ISODate("2022-08-01T00:00"),
"completedOn": ISODate("2022-07-24T00:00")
}
For example, if I group by created date, the record for July 2022 would show 1 created job and only 1 completed job, but it should show 2.
How can I go about recreating this report? One idea was that I needed to determine the minimum and maximum of all the possible dates across those 3 date fields in my collection, but I don't know where to go from there
I ended up solving this by using a facet. I followed this process:
Each facet field grouped by a different date field from the source document, and then aggregated the relevant field (e.g. counts, or sums as required). I ensured each of these fields in the facet had a unique name.
I then did a project stage where I took each of the facet stage fields (arrays), and concat them into a single array
I unwound the array, and then replaced the root to make it simpler to work with
I then grouped again by the _id field which was set to the relevant date during the facet field, and then grabbed the relevant fields.
The relevant parts of the pipeline are below:
db.getCollection("jobs").aggregate(
// Pipeline
[
// Stage 3
{
$facet: {
//Facet 1, group by created date, count number of jobs created
//facet 2, group by completed date, count number of jobs completed
//facet 3, group by due date, count number of jobs due
"created" : [
{
$addFields : {
"monthStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$createdAt",
"unit" : "month",
"binSize" : 1.0,
"timezone" : "$timezone",
"startOfWeek" : "mon"
}
},
"timezone" : "$timezone"
}
}
}
},
"yearStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$createdAt",
"unit" : "year",
"binSize" : 1.0,
"timezone" : "$timezone"
}
},
"timezone" : "$timezone"
}
}
}
}
}
},
{
$group : {
"_id" : {
"year" : "$yearStarting",
"month" : "$monthStarting"
},
"monthStarting" : {
"$first" : "$monthStarting"
},
"yearStarting" : {
"$first" : "$yearStarting"
},
"createdCount": {$sum: 1}
}
}
],
"completed" : [
{
$addFields : {
"monthStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$completedDate",
"unit" : "month",
"binSize" : 1.0,
"timezone" : "$timezone",
"startOfWeek" : "mon"
}
},
"timezone" : "$timezone"
}
}
}
},
"yearStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$completedDate",
"unit" : "year",
"binSize" : 1.0,
"timezone" : "$timezone"
}
},
"timezone" : "$timezone"
}
}
}
}
}
},
{
$group : {
"_id" : {
"year" : "$yearStarting",
"month" : "$monthStarting"
},
"monthStarting" : {
"$first" : "$monthStarting"
},
"yearStarting" : {
"$first" : "$yearStarting"
},
"completedCount": {$sum: 1}
}
}
],
"due": [
{
$match: {
"dueDate": {$ne: null}
}
},
{
$addFields : {
"monthStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$dueDate",
"unit" : "month",
"binSize" : 1.0,
"timezone" : "$timezone",
"startOfWeek" : "mon"
}
},
"timezone" : "$timezone"
}
}
}
},
"yearStarting" : {
"$dateFromString" : {
"dateString" : {
"$dateToString" : {
"date" : {
"$dateTrunc" : {
"date" : "$dueDate",
"unit" : "year",
"binSize" : 1.0,
"timezone" : "$timezone"
}
},
"timezone" : "$timezone"
}
}
}
}
}
},
{
$group : {
"_id" : {
"year" : "$yearStarting",
"month" : "$monthStarting"
},
"monthStarting" : {
"$first" : "$monthStarting"
},
"yearStarting" : {
"$first" : "$yearStarting"
},
"dueCount": {$sum: 1},
"salesRevenue": {$sum: "$totalSellPrice"},
"costGenerated": {$sum: "$totalBuyPrice"},
"profit": {$sum: "$profit"},
"avgValue": {$avg: "$totalSellPrice"},
"finalisedRevenue": {$sum: {
$cond: {
"if": {$in: ["$status",["Finalised","Closed"]]},
"then": "$totalSellPrice",
"else": 0
}
}}
}
}
]
}
},
// Stage 4
{
$project: {
"docs": {$concatArrays: ["$created","$completed","$due"]}
}
},
// Stage 5
{
$unwind: {
path: "$docs",
}
},
// Stage 6
{
$replaceRoot: {
// specifications
"newRoot": "$docs"
}
},
// Stage 7
{
$group: {
_id: "$_id",
"monthStarting" : {
"$first" : "$monthStarting"
},
"yearStarting" : {
"$first" : "$yearStarting"
},
"monthStarting" : {
"$first" : "$monthStarting"
},
"createdCountSum" : {
"$sum" : "$createdCount"
},
"completedCountSum" : {
"$sum" : "$completedCount"
},
"dueCountSum" : {
"$sum" : "$dueCount"
},
"salesRevenue" : {
"$sum" : "$salesRevenue"
},
"costGenerated" : {
"$sum" : "$costGenerated"
},
"profit" : {
"$sum" : "$profit"
},
"finalisedRevenue" : {
"$sum" : "$finalisedRevenue"
},
"avgJobValue": {
$sum: "$avgValue"
}
}
},
],
);

Sort documents by value in the last element of an array that matches filter. Mongodb

I have a collection of evaluationGroups with the following documents structure:
{
"_id" : ObjectId("60073749694fd4d81e4d677d"),
"AlertSettingId" : ObjectId("5ffddaaa0b1d2c30b191599a"),
"CurrentStatus" : "success",
"Evaluations" : [
{
"EvaluatedAt" : ISODate("2021-01-19T19:47:18.850Z"),
"ReferenceValue" : 1.0,
"Status" : "success"
},
{
"EvaluatedAt" : ISODate("2021-01-19T19:52:16.423Z"),
"ReferenceValue" : 1.0,
"Status" : "triggered"
},
{
"EvaluatedAt" : ISODate("2021-01-19T21:47:16.400Z"),
"ReferenceValue" : 1.0,
"Status" : "success"
}
]
}
{
"_id" : ObjectId("60085ec60a264ce3829a6335"),
"AlertSettingId" : ObjectId("5ffddaaa0b1d2c30b191599a"),
"CurrentStatus" : "triggered",
"Evaluations" : [
{
"EvaluatedAt" : ISODate("2021-01-20T18:03:01.040Z"),
"ReferenceValue" : 1.0,
"Status" : "noDataFound"
},
{
"EvaluatedAt" : ISODate("2021-01-20T22:04:43.983Z"),
"ReferenceValue" : 1.0,
"Status" : "triggered"
},
{
"EvaluatedAt" : ISODate("2021-01-20T22:39:43.978Z"),
"ReferenceValue" : 1.0,
"Status" : "triggered"
},
]
}
{
"_id" : ObjectId("60099092f7386972de3e8a05"),
"AlertSettingId" : ObjectId("5ffddaaa0b1d2c30b191599a"),
"CurrentStatus" : "success",
"Evaluations" : [
{
"EvaluatedAt" : ISODate("2021-01-21T14:32:48.697Z"),
"ReferenceValue" : 1.0,
"Status" : "noDataFound"
},
{
"EvaluatedAt" : ISODate("2021-01-21T14:37:44.929Z"),
"ReferenceValue" : 1.0,
"Status" : "triggered"
},
{
"EvaluatedAt" : ISODate("2021-01-21T14:42:44.928Z"),
"ReferenceValue" : 1.0,
"Status" : "triggered"
},
{
"EvaluatedAt" : ISODate("2021-01-21T15:17:46.052Z"),
"ReferenceValue" : 1.0,
"Status" : "success"
}
]
}
What I need to do is to sort all evaluation groups by the latest evaluation inside Evaluations (using EvaluatedAt property as the sort), but that the evaluation has also status triggered.
So, to sum up, I have to sort the groups, by the latest triggered Evaluation date.
I was looking at the question: Mongodb: sort documents by value in the last element of an array
And I liked this response of how to sort by last item, (because latest evaluations are at the end of the array in my case):
db.collection.aggregate([
{
$addFields: {
lastSent: {
$let: {
vars: {
last: {
$arrayElemAt: [ "$messages", -1 ]
}
},
in: "$$last.commData.sent.dateTime"
}
}
}
},
{ $sort: { lastSent: 1 } },
{ $project: { lastSent: 0 } }
])
But I would need to also filter evaluations by status "triggered" before getting the latest one.
How can achieve this using MongoDB aggregate query?
You can use $filter operator,
$filter to filter Evaluations array on the base of Status
$max to get latest EvaluatedAt form filtered result
db.collection.aggregate([
{
$addFields: {
lastSent: {
$let: {
vars: {
filtered: {
$filter: {
input: "$Evaluations",
cond: { $eq: ["$$this.Status", "triggered"] }
}
}
},
in: { $max: "$$filtered.EvaluatedAt" }
}
}
}
},
{ $sort: { lastSent: 1 } },
{ $project: { lastSent: 0 } }
])
Playground

MongoDb Group BY MAX date and get field from this document

Look my problen hehe. I don't know how I can do it.
There are a lot of assetState 1..n I would like do aggregation for get last asset state group by asset.
Mongo collection : assetState
[
{
"lsd" : {
"$id" : ObjectId("lucas")
},
"stateDate" : ISODate("2018-09-10T16:26:44.501Z"),
"assetId" : ObjectId("5b96b7645f2b3c0101520s60")
},
{
"lsd" : {
"$id" : ObjectId("denner")
},
"stateDate" : ISODate("2018-09-10T17:26:44.501Z"),
"assetId" : ObjectId("5b96b7645f2b3c0101520s60")
},
{
"lsd" : {
"$id" : ObjectId("denner")
},
"stateDate" : ISODate("2018-09-10T18:26:44.501Z"),
"assetId" : ObjectId("5b96b7645f2a8c0001530f61")
},
{
"lsd" : {
"$id" : ObjectId("lermen")
}
},
"stateDate" : ISODate("2018-09-10T20:26:44.501Z"),
"assetId" : ObjectId("5b96b7645f2a8c0001530f61")
},
{
"lsd" : {
"$id" : ObjectId("floripa")
},
"stateDate" : ISODate("2018-09-10T19:26:44.501Z"),
"assetId" : ObjectId("5b96b7645f2a8c0001530f61")
}
]
I would like get max "stateDate", so I need get LSD from same row(document).
Expected result:
{
"lsd" : {
"$id" : ObjectId("lermen")
},
"stateDate" : ISODate("2018-09-10T20:26:44.501Z")
}
I tried to do:
db.getCollection('assetState').aggregate([
{
$group: {
"_id": {"assetId": "$assetId"},
"stateDate": {
"$max": "$stateDate"
},
"lsd": {$last: "$lsd"} // I tried change $max to $min and $last it din't work :(
}
]);
Result:
{
"lsd" : {
"$id" : ObjectId("floripa")
},
"stateDate" : ISODate("2018-09-10T20:26:44.501Z")
}
Many Thanks
Try this query
db.getCollection('assetState').aggregate([
{$sort:{"stateDate":-1}},
]).limit(1)
you can $sort descending before $group and gets the first item of each group with $arrayElemAt
db.getCollection('assetState').aggregate([
{ $sort: { stateDate: -1 } },
{ $group: { _id: { "assetId" : "$assetId" },
states: { $push: "$$ROOT" }
}
},
{ $project: { "last_asset": { $arrayElemAt: [ "$states", 0 ] }, _id:0 } },
])
Result:
/* 1 */
{
"last_asset" : {
"_id" : ObjectId("5db2b34fa1b70230bba9c4d9"),
"lsd" : "denner",
"stateDate" : ISODate("2018-09-10T17:26:44.501Z"),
"assetId" : "5b96b7645f2b3c0101520s60"
}
}
/* 2 */
{
"last_asset" : {
"_id" : ObjectId("5db2b34fa1b70230bba9c4db"),
"lsd" : "lermen",
"stateDate" : ISODate("2018-09-10T20:26:44.501Z"),
"assetId" : "5b96b7645f2a8c0001530f61"
}
}
You could use $unwind (aggregation)
https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/
I have this for the group by phase:
{
_id: {"sourceOfEvent":"$sourceOfEvent","vehicleId":"$vehicleId"},
maxTimeOfEvent: { $max: "$timeOfEvent" }
}
So group by on 2 fields and I use the max to get the max of the timeOfEvent (which is a date)

Mongodb How to pivot fields?

I have collection in mongo db,
so I want the fields: Ip UtilizationReports.Start UtilizationReports.CpuReports.0.ProcName UtilizationReports.CpuReports.0.MaxUsage UtilizationReports.CpuReports.1.ProcName UtilizationReports.CpuReports.1.MaxUsage
for that I use this this code:
db.Agents.aggregate(
[
{
"$project" : {
"UtilizationReports.Start" : 0.0,
"UtilizationReports.Interval" : 0.0,
"UtilizationReports.LastLogin" : 0.0,
"Configuration" : 0.0,
"HealthReports" : 0.0,
"_id" : 0.0
}
},
{
"$unwind" : {
"path" : "$UtilizationReports.CpuReports"
}
},
{
"$match" : {
"UtilizationReports.CpuReports.0" : {
"$exists" : true
},
"UtilizationReports.0.CpuReports.0.ProcName" : {
"$exists" : true
}
}
}
],
{
"allowDiskUse" : false
}
);
and I get this data
but I want to pivot row "ProcName" to column and the value will be "MaxUsage"
I try this code but it doesn't work:
db.Agents.aggregate(
[
{
"$project" : {
"tmp" : {
"$arrayToObject" : {
"$zip" : {
"inputs" : [
"$items.key",
"$items.value"
]
}
}
}
}
},
{
"$group" : {
"_id" : {
"Ip" : "$Ip",
"StartReport" : "$UtilizationReports.Start"
},
"items" : {
"$addToSet" : {
"key" : "$UtilizationReports.CpuReports.ProcName",
"value" : "$UtilizationReports.CpuReports.MaxUsage"
}
}
}
},
{
"$addFields" : {
"tmp.Ip" : "$_id.Ip",
"tmp.StartReport" : "$_id.StartReport"
}
},
{
"$replaceRoot" : {
"newRoot" : "$tmp"
}
}
],
{
"allowDiskUse" : true
}
);
the error message:
ad projection specification, cannot include fields or add computed fields during an exclusion projection: { UtilizationReports.Interval: 0.0, UtilizationReports.LastLogin: 0.0, Configuration: 0.0, HealthReports: 0.0, _id: 0.0, tmp: { $arrayToObject: { $zip: { inputs: [ \"$items.key\", \"$items.value\" ]
The sample output:
$arrayToObject is New in version 3.6.
I use with mongoDB version 3.4

Mongodb Aggregation Pipeline Count Total size across multiple fields

`"ActivityScores" : {
"Spring" : [
{
"ActivityId" : "8fd38724-7e7d-4518-bd49-d38a8b4b3435",
"ActivityTime" : "2017-05-25T16:07:02.000-06:00"
}
],
"Winter" : [
{
"ActivityId" : "90d2a976-19d9-4ce0-aa88-d32c122d173b",
"ActivityTime" : "2017-02-14T22:50:00.000-06:00"
}
],
"Fall" : [
{
"ActivityId" : "84b8c41e-788f-4acd-abec-dc455285972b",
"ActivityTime" : "2016-11-15T22:37:02.000-06:00"
},
{
"ActivityId" : "157af880-d47b-42fc-8ecf-ecfc1bbb56b1",
"ActivityTime" : "2016-09-01T22:50:05.000-06:00"
}
]
},
"Grade" : "2",
"GradeTag" : "GRADE_2", `
I am looking for aggregation query to get Total of ActivityIds. I tried various combination of $group, $unwind, $size $addToset but none of them seems to be working . I need to find total activities using aggregation framework only. I don't want to go through each document using javascript or python to get the total counts. Is there any easy way around?
Thanks.We are on version 3.2.Finally below combination worked. ActivityScores was field to entity.SchoolYears in our Schema.Working Aggregation Pipeline for me.
db.studentcontentareadocument.aggregate(
[
{
$project: {
"SpringActivitiesPerDoc" : {
"$size" : "$entity.SchoolYears.ActivityScores.Spring"
},
"WinterActivitiesPerDoc" : {
"$size" : "$entity.SchoolYears.ActivityScores.Winter"
},
"FallActivitiesPerDoc" : {
"$size" : "$entity.SchoolYears.ActivityScores.Fall"
}
}
},
{
$project: {
"TotalActivitiesPerDoc" : {
"$add" : [
"$SpringActivitiesPerDoc",
"$WinterActivitiesPerDoc",
"$FallActivitiesPerDoc"
]
}
}
},
{
$group: {
"_id" : null,
"TotalActivities" : {
"$sum" : "$TotalActivitiesPerDoc"
}
}
},
{
$project: {
"_id" : 0,
"TotalSGPActivities" : "$TotalActivities"
}
}
],
{
cursor: {
batchSize: 50
},
allowDiskUse: true
}
);