How can I do match after second level unwind in mongodb? - mongodb

I am working on a software that uses MongoDB as a database. I have a collection like this (this is just one document)
{
"_id" : ObjectId("5aef51e0af42ea1b70d0c4dc"),
"EndpointId" : "89799bcc-e86f-4c8a-b340-8b5ed53caf83",
"DateTime" : ISODate("2018-05-06T19:05:04.574Z"),
"Url" : "test",
"Tags" : [
{
"Uid" : "E2:02:00:18:DA:40",
"Type" : 1,
"DateTime" : ISODate("2018-05-06T19:05:04.574Z"),
"Sensors" : [
{
"Type" : 1,
"Value" : NumberDecimal("-98")
},
{
"Type" : 2,
"Value" : NumberDecimal("-65")
}
]
},
{
"Uid" : "12:3B:6A:1A:B7:F9",
"Type" : 1,
"DateTime" : ISODate("2018-05-06T19:05:04.574Z"),
"Sensors" : [
{
"Type" : 1,
"Value" : NumberDecimal("-95")
},
{
"Type" : 2,
"Value" : NumberDecimal("-59")
},
{
"Type" : 3,
"Value" : NumberDecimal("12.939770381907275")
}
]
}
]
}
and I want to run this query on it.
db.myCollection.aggregate([
{ $unwind: "$Tags" },
{
$match: {
$and: [
{
"Tags.DateTime": {
$gte: ISODate("2018-05-06T19:05:02Z"),
$lte: ISODate("2018-05-06T19:05:09Z"),
},
},
{ "Tags.Uid": { $in: ["C1:3D:CA:D4:45:11"] } },
],
},
},
{ $unwind: "$Tags.Sensors" },
{ $match: { "$Tags.Sensors.Type": { $in: [1, 2] } } },
{
$project: {
_id: 0,
EndpointId: "$EndpointId",
TagId: "$Tags.Uid",
Url: "$Url",
TagType: "$Tags.Type",
Date: "$Tags.DateTime",
SensorType: "$Tags.Sensors.Type",
Value: "$Tags.Sensors.Value",
},
},
])
the problem is, the second match (that checks $Tags.Sensors.Type) doesn't work and doesn't affect the result of the query.
How can I solve that?
If this is not the right way, what is the right way to run these conditions?

The $match stage accepts field names without a leading $ sign. You've done that correctly in your first $match stage but in the second one you write $Tags.Sensors.Type. Simply removing the leading $ sign should make your query work.
Mind you, the whole thing can be a bit simplified (and some beautification doesn't hurt, either):
You don't need to use $and in your example since it's assumed by default if you specify more than one criterion in a filter.
The $in that you use for the Tags.Sensors.Type filter can be a simple : kind of equality operator unless you have more than one element in the list of acceptable values.
In the $project stage, instead of (kind of) duplicating identical field names you can use the <field>: 1 syntax unless the order of the fields matters.
So the final query would be something like this.
db.myCollection.aggregate([
{
"$unwind" : "$Tags"
},
{
"$match" : {
"Tags.DateTime" : { "$gte" : ISODate("2018-05-06T19:05:02Z"), "$lte" : ISODate("2018-05-06T19:05:09Z") },
"Tags.Uid" : { "$in" : ["C1:3D:CA:D4:45:11"] }
}
}, {
"$unwind" : "$Tags.Sensors"
}, {
"$match" : {
"Tags.Sensors.Type" : { "$in" : [1,2] }
}
},
{
"$project" : {
"_id" : 0,
"EndpointId" : 1,
"TagId" : "$Tags.Uid",
"Url" : 1,
"TagType" : "$Tags.Type",
"Date" : "$Tags.DateTime",
"SensorType" : "$Tags.Sensors.Type",
"Value" : "$Tags.Sensors.Value"
}
}])

Related

Mongodb Query to get the nth document

I need to create a query in mongodb that needs to return the SECOND TO THE LAST document. I am planning to use $group for this query but i dont know what aggregation function to use. I only know $first and $last.
I have an example collection below and also include the expected output. Thank you!
"_id" : ObjectId("60dc27ac54b7c46bfa1b84b4"),
"auditlogs" : [
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84be"),
"userid" : ObjectId("5ffe702d59a9205db81fcb69"),
"action" : "ADDTRANSACTION"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84bd"),
"userid" : ObjectId("5ffe644f9493e05db9245192"),
"action" : "EDITPROFILE"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84bc"),
"userid" : ObjectId("5ffe64949493e05db9245197"),
"action" : "DELETETRANSACTION"
} ]
"_id" : ObjectId("60dc27ac54b7c46bfa1b75ge2"),
"auditlogs" : [
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84bb"),
"userid" : ObjectId("5ffe64b69493e05db924519b"),
"action" : "ADDTRANSACTION"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84ba"),
"userid" : ObjectId("5ffe65419493e05db92451d4"),
"action" : "ADDTRANSACTION"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84b9"),
"userid" : ObjectId("5ffe65689493e05db92451d9"),
"action" : "CHANGEACCESS"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84b8"),
"userid" : ObjectId("5ffe65819493e05db92451dd"),
"action" : "DELETETRANSACTION"
},
{
"_id" : ObjectId("60dc27ac54b7c46bfa1b84b7"),
"userid" : ObjectId("5ffe65df9493e05db92451f3"),
"action" : "EDITPROFILE",
]
OUTPUT:
{"_id" : ObjectId("60dc27ac54b7c46bfa1b84b4"),"_id" : ObjectId("60dc27ac54b7c46bfa1b84bd"),"userid" : ObjectId("5ffe644f9493e05db9245192"),"action" : "EDITPROFILE"},
{"_id" : ObjectId("60dc27ac54b7c46bfa1b75ge2"),"_id" : ObjectId("60dc27ac54b7c46bfa1b84b8"),"userid" : ObjectId("5ffe65819493e05db92451dd"),"action" : "DELETETRANSACTION"}
You can't have two _id keys in one single object.
I've made the parent object's id to _parentId you can give it's a name anything you want except _id
Aggregation:
db.collection.aggregate([
{
$unwind: "$auditlogs"
},
{
"$project": {
"_parentId": "$_id",
"_id": "$auditlogs._id",
"action": "$auditlogs.action",
"userid": "$auditlogs.userid",
}
}
])
Playground
You can slice the array by -2 to get the last two item, then by 1 to get first one. Therefore, the array will be left the second to the last. Finally, unwind auditlogs so it can be changed from array to object which is structure that you want.
db.collection.aggregate([
{
$project: { auditlogs : { $slice: [ "$auditlogs", -2 ] } }
},
{
$project: { auditlogs : { $slice: [ "$auditlogs", 1 ] } }
},
{
$unwind: "$auditlogs"
}
])

Slow mongodb query, how can it happen?

I have the following query in MongoDB:
db.getCollection('message').aggregate([
{
"$match": {
"who" : { "$in" : ["manager", "woker"] },
"sendTo": { "$in": ["userId:243369", "userId:160921"] },
"exceptSendTo": { "$nin": ["userId:37355"] },
"msgTime": { "$lt": 1559716155 },
"isInvalid": { "$exists": false }
}
},
{
"$sort": { "msgTime": 1, "who": 1, "sendTo": 1 }
},
{
"$group": { "_id": "$who", "doc": { "$first": "$type" } }
}
], { allowDiskUse: true})
forget about the field meaning. and I have this index:
/* 1 */
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "db.message"
},
{
"v" : 1,
"key" : {
"who" : 1.0,
"sendTo" : 1.0
},
"name" : "who_sendTo",
"ns" : "db.message"
},
{
"v" : 1,
"key" : {
"msgTime" : 1.0
},
"name" : "msgTime_1",
"ns" : "db.message"
},
{
"v" : 1,
"key" : {
"msgTime" : 1.0,
"who" : 1.0,
"sendTo" : 1.0
},
"name" : "msgTime_1.0_who_1.0_sendTo_1.0",
"ns" : "db.message",
"background" : true
}
]
Perform the query above, It cost 1.52s, use explain to see it indeed has used msgTime_1.0_who_1.0_sendTo_1.0 index.
Why is query is still low while index has been used? and is there any way to solve the low problem like change index or something?
I dont think you are using the sort at all the way you intend to use it.
The $firs argument requires a sort on the actual first arguement
https://docs.mongodb.com/manual/reference/operator/aggregation/first/
You need to sort the key you want the first element of.
OR you could use $$ROOT, witch returns the first document.
I think you should modify it to something like:
{"$sort": {"who": 1, "msgTime": 1, "sendTo": 1}},
{"$group": {"_id": "$who", "doc": {"$first": "$$root"}}},
In this case the $group operator can find the result for each group "instantly" since they are all next to each other.
If you are only interested in the type, add an projection:
{'$project': {'doc.type': 1}

Sort a match group by id in aggregate

(Mongo newbie here, sorry) I have a mongodb collection, result of a mapreduce with this schema :
{
"_id" : "John Snow",
"value" : {
"countTot" : 500,
"countCall" : 30,
"comment" : [
{
"text" : "this is a text",
"date" : 2016-11-17 00:00:00.000Z,
"type" : "call"
},
{
"text" : "this is a text",
"date" : 2016-11-12 00:00:00.000Z,
"type" : "visit"
},
...
]
}
}
My goal is to have a document containing all the comments of a certain type. For example, a document John snow with all the calls.
I manage to have all the comments for a certain type using this :
db.general_stats.aggregate(
{ $unwind: '$value.comment' },
{ $match: {
'value.comment.type': 'call'
}}
)
However, I can't find a way to group the data received by the ID (for example john snow) even using the $group property. Any idea ?
Thanks for reading.
Here is the solution for your query.
db.getCollection('calls').aggregate([
{ $unwind: '$value.comment' },
{ $match: {
'value.comment.type': 'call'
}},
{
$group : {
_id : "$_id",
comment : { $push : "$value.comment"},
countTot : {$first : "$value.countTot"},
countCall : {$first : "$value.countCall"},
}
},
{
$project : {
_id : 1,
value : {"countTot":"$countTot","countCall":"$countCall","comment":"$comment"}
}
}
])
or either you can go with $project with $filter option
db.getCollection('calls').aggregate([
{
$project: {
"value.comment": {
$filter: {
input: "$value.comment",
as: "comment",
cond: { $eq: [ "$$comment.type", 'call' ] }
}
},
"value.countTot":"$value.countTot",
"value.countCall":"$value.countCall",
}
}
])
In both case below is my output.
{
"_id" : "John Snow",
"value" : {
"countTot" : 500,
"countCall" : 30,
"comment" : [
{
"text" : "this is a text",
"date" : "2016-11-17 00:00:00.000Z",
"type" : "call"
},
{
"text" : "this is a text 2",
"date" : "2016-11-17 00:00:00.000Z",
"type" : "call"
}
]
}
}
Here is the query which is the extension of the one present in OP.
db.general_stats.aggregate(
{ $unwind: '$value.comment' },
{ $match: {
'value.comment.type': 'call'
}},
{$group : {_id : "$_id", allValues : {"$push" : "$$ROOT"}}},
{$project : {"allValues" : 1, _id : 0} },
{$unwind : "$allValues" }
);
Output:-
{
"allValues" : {
"_id" : "John Snow",
"value" : {
"countTot" : 500,
"countCall" : 30,
"comment" : {
"text" : "this is a text",
"date" : ISODate("2016-11-25T10:46:49.258Z"),
"type" : "call"
}
}
}
}
Got my answer looking at this :
How to retrieve all matching elements present inside array in Mongo DB?
using the $addToSet property in the $group one.

Multiplication on nested fields on MongoDB

In a database in MongoDB I am trying to group some data by their date (one group for each day of the year), and then add an additional field that would be the result of the multiplication of two of the already existing fields.
The data structure is:
{
"_id" : ObjectId("567a7c6d9da4bc18967a3947"),
"units" : 3.0,
"price" : 50.0,
"name" : "Name goes here",
"datetime" : ISODate("2015-12-23T10:50:21.560+0000")
}
I first tried a two stage approach using $project and then $group like this
db.things.aggregate(
[
{
$project: {
"_id" : 1,
"name" : 1,
"units" : 1,
"price" : 1,
"datetime":1,
"unitsprice" : { $multiply: [ "$price", "$units" ] }
}
},
{
$group: {
"_id" : {
"day" : {
"$dayOfMonth" : "$datetime"
},
"month" : {
"$month" : "$datetime"
},
"year" : {
"$year" : "$datetime"
}
},
"things" : {
"$push" : "$$ROOT"
}
}
}
],
)
in this case, the first step (the $project) gives the expected output (with the expected value of unitsprice), but then when doing the second $group step, it outputs this error:
"errmsg":$multiply only supports numeric types, not String",
"code":16555
I tried also turning around things, doing the $group step first and then the $project
db.things.aggregate(
[
{
$group: {
"_id" : {
"day" : {
"$dayOfMonth" : "$datetime"
},
"month" : {
"$month" : "$datetime"
},
"year" : {
"$year" : "$datetime"
}
},
"things" : {
"$push" : "$$ROOT"
}
}
},
{
$project: {
"_id" : 1,
"things":{
"name" : 1,
"units" : 1,
"price" : 1,
"datetime":1,
"unitsprice" : { $multiply: [ "$price", "$units" ] }
}
}
}
],
);
But in this case, the result of the multiplication is: unitsprice:null
Is there any way of doing this multiplication? Also, it would be nice to do it in a way that the output would not have nested fields, so it would look like:
{"_id":
"units":
"price":
"name":
"datetime":
"unitsprice":
}
Thanks in advance
PS:I am running MongoDB 3.2
Finally found the error. When importing one of the fields, a few of the price fields were created as a string. Surprisingly, the error didn't came out when first doing the multiplication in the project step (the output was normal until it reached the first wrong field, then it stopped), but when doing the group step.
In order to find the text fields I used this query:
db.things.find( { price: { $type: 2 } } );
Thanks for the hints

MongoDB $sum and $avg of sub documents

I need to get $sum and $avg of subdocuments, i would like to get $sum and $avg of Channels[0].. and other channels as well.
my data structure looks like this
{
_id : ... Location : 1,
Channels : [
{ _id: ...,
Value: 25
},
{
_id: ... ,
Value: 39
},
{
_id: ..,
Value: 12
}
]
}
In order to get the sum and average of the Channels.Value elements for each document in your collection you will need to use mongodb's Aggregation processing. Further, since Channels is an array you will need to use the $unwind operator to deconstruct the array.
Assuming that your collection is called example, here's how you could get both the document sum and average of the Channels.Values:
db.example.aggregate( [
{
"$unwind" : "$Channels"
},
{
"$group" : {
"_id" : "$_id",
"documentSum" : { "$sum" : "$Channels.Value" },
"documentAvg" : { "$avg" : "$Channels.Value" }
}
}
] )
The output from your post's data would be:
{
"_id" : SomeObjectIdValue,
"documentSum" : 76,
"documentAvg" : 25.333333333333332
}
If you have more than one document in your collection then you will see a result row for each document containing a Channels array.
Solution 1: Using two groups based this example:
previous question
db.records.aggregate(
[
{ $unwind: "$Channels" },
{ $group: {
_id: {
"loc" : "$Location",
"cId" : "$Channels.Id"
},
"value" : {$sum : "$Channels.Value" },
"average" : {$avg : "$Channels.Value"},
"maximun" : {$max : "$Channels.Value"},
"minimum" : {$min : "$Channels.Value"}
}},
{ $group: {
_id : "$_id.loc",
"ChannelsSumary" : { $push :
{ "channelId" : '$_id.cId',
"value" :'$value',
"average" : '$average',
"maximun" : '$maximun',
"minimum" : '$minimum'
}}
}
}
]
)
Solution 2:
there is property i didn't show on my original question that might of help "Channels.Id" independent from "Channels._Id"
db.records.aggregate( [
{
"$unwind" : "$Channels"
},
{
"$group" : {
"_id" : "$Channels.Id",
"documentSum" : { "$sum" : "$Channels.Value" },
"documentAvg" : { "$avg" : "$Channels.Value" }
}
}
] )