add where condition in aggregate and group function in mongodb - mongodb

I have mongo model lets say MYLIST containing data like:-
{
"_id" : ObjectId("542139f31284ad1461dbc15f"),
"Category" : "CENTER",
"Name" : "STAND",
"Url" : "center/stand",
"Img" : [ {
"url" : "www.google.com/images",
"main" : "1",
"home" : "1",
"id" : "34faf230-43cf-11e4-8743-311ea2261289"
},
{
"url" : "www.google.com/images1",
"main" : "1",
"home" : "0",
"id" : "34faf230-43cf-11e4-8743-311e66441289"
} ]
}
I execute the following query to the MYLIST collection:
db.MYLIST.aggregate([
{ "$group": {
"_id": "$Category",
"Name": { "$addToSet": {
"name": "$Name",
"url": "$Url",
"img": "$Img"
}}
}},
{ "$sort": { "_id" : 1 } }
]);
And I got the following result -
[
{ _id: 'CENTER',
Name:
[ { "name" : "Stand",
"url" : "center/stand",
"img": { "url" : "www.google.com/images" , "main" : "1", "home" : "1", "id" : "350356a0-43cf-11e4-8743-311ea2261289" }
}]
},
{ _id: 'CENTER',
Name:
[ { "name" : "Stand",
"url" : "center/stand",
"img": { "url" : "www.google.com/images1" , "main" : "1", "home" : "0", "id" : "34faf230-43cf-11e4-8743-311ea2261289" }
}]
}
]
As you can see my img key itself is an array of objects, Hence I am getting multiple entries for the same category of each entry in img array.
What I actually need is to get only those images that have some value for home key.
expected result:-
[
{ _id: 'CENTER',
Name:
[ { "name" : "Stand",
"url" : "center/stand",
"img": { "url" : "www.google.com/images" , "main" : "1", "home" : "1", "id" : "350356a0-43cf-11e4-8743-311ea2261289" }
}]
},
]
Hence I would like to add where the condition for img.home > 0 on the above-mentioned query, Could anybody help me to resolve this issue as my relatively new to MongoDB.

Still really not sure if this is what you want or even why you would be using $addToSet on this grouping. But if all you want to do is "filter" the content of the array returned in your result, then what you want to do is $match the array elements to your condition after processing an $unwind pipeline in order to "de-normalize" the content:
db.MYLIST.aggregate([
// If you only want those matching array members it makes sense to match the
// documents that contain them first
{ "$match": { "Img.home": 1 } },
// Unwind to de-normalize or "un-join" the documents
{ "$unwind": "$Img" },
// Match again to "filter" out those elements that do not match
{ "$match": { "Img.home": 1 } },
// Then do your grouping
{ "$group": {
"_id": "$Category",
"Name": {
"$addToSet": {
"name": "$Name",
"url": "$Url",
"img": "$Img"
}
}
}},
// Finally sort
{ "$sort": { "_id" : 1 } }
]);
So the $match pipeline is the equivalent of a general query or "where clause" in SQL terms, and can be used at any stage. It is usually best to have this as a first stage when there is some type of filtering that results from this. It reduces the overall load by reducing documents to be processed even if "all" of the end results are not removed as would be the case of working with an array.
The $unwind stage allows the array elements to be processed just like another document. And of course you can just use another $match pipeline stage after this in order to just match the documents to your query condition.

Related

MongoDB query to get last record of each id

I want to get last record of each sender.id based on createdAt using mongodb query.
Sample json:
{
"code" : "34242342",
"name" : "name1",
"amount" : 200,
"sender" : {
"id" : "fsrfsr3242",
"name" : "name2",
"phone" : "12345678",
"category": "cat1"
},
"receiver" : {
"id" : "42342rewr",
"name" : "naem3",
"phone" : "5653679755"
},
"message" : "",
"status" : "done",
"createdAt" : ISODate("2019-09-27T09:17:32.597Z")
}
Query i tried:
[{
$match: {
'sender.category': "cat1"
}
}, {
$group: {
_id: "$sender.id",
lastrecord: {
$last: "$createdAt"
}
}
}]
I want to return entire json as above with only last record of each sender.id. Above query is only giving me only last date , How do i return entire json using aggregation pipeline?
I am using groupby because each sender.id can have multiple records of which i only want to retrieve the last one.
You can use $$ROOT variable to get the whole last document
[
{ "$match": { "sender.category": "cat1" }},
{ "$sort": { "$createdAt": 1 }},
{ "$group": {
"_id": "$sender.id",
"lastrecord": {
"$last": "$$ROOT"
}
}}
]

Multiple Nested Group Within Array

I'm having group of elements in MongoDB as given below:
/* 1 */
{
"_id" : ObjectId("58736c7f7d43c305461cdb9b"),
"Name" : "Kevin",
"pb_event" : [
{
"event_type" : "Birthday",
"event_date" : "2014-08-31"
},
{
"event_type" : "Anniversary",
"event_date" : "2014-08-31"
}
]
}
/* 2 */
{
"_id" : ObjectId("58736cfc7d43c305461cdba8"),
"Name" : "Peter",
"pb_event" : [
{
"event_type" : "Birthday",
"event_date" : "2014-08-31"
},
{
"event_type" : "Anniversary",
"event_date" : "2015-03-24"
}
]
}
/* 3 */
{
"_id" : ObjectId("58736cfc7d43c305461cdba9"),
"Name" : "Pole",
"pb_event" : [
{
"event_type" : "Birthday",
"event_date" : "2015-03-24"
},
{
"event_type" : "Work Anniversary",
"event_date" : "2015-03-24"
}
]
}
Now I want the result that has group on event_date then after group on event_type. event_type contain all names of the related user, then count of records in the respective array.
Expected Output
/* 1 */
{
"event_date" : "2014-08-31",
"data" : [
{
"event_type" : "Birthday",
"details" : [
{
"_id" : ObjectId("58736c7f7d43c305461cdb9b"),
"name" : "Kevin"
},
{
"_id" : ObjectId("58736cfc7d43c305461cdba8"),
"name" : "Peter"
}
],
"count" : 2
},
{
"event_type" : "Anniversary",
"details" : [
{
"_id" : ObjectId("58736c7f7d43c305461cdb9b"),
"name" : "Kevin"
}
],
"count" : 1
}
]
}
/* 2 */
{
"event_date" : "2015-03-24",
"data" : [
{
"event_type" : "Anniversary",
"details" : [
{
"_id" : ObjectId("58736cfc7d43c305461cdba8"),
"name" : "Peter"
}
],
"count" : 1
},
{
"event_type" : "Birthday",
"details" : [
{
"_id" : ObjectId("58736cfc7d43c305461cdba9"),
"name" : "Pole"
}
],
"count" : 1
},
{
"event_type" : "Work Anniversary",
"details" : [
{
"_id" : ObjectId("58736cfc7d43c305461cdba9"),
"name" : "Pole"
}
],
"count" : 1
}
]
}
Using the aggregation framework, you would need to run a pipeline that has the following stages so that you get the desired result:
db.collection.aggregate([
{ "$unwind": "$pb_event" },
{
"$group": {
"_id": {
"event_date": "$pb_event.event_date",
"event_type": "$pb_event.event_type"
},
"details": {
"$push": {
"_id": "$_id",
"name": "$Name"
}
},
"count": { "$sum": 1 }
}
},
{
"$group": {
"_id": "$_id.event_date",
"data": {
"$push": {
"event_type": "$_id.event_type",
"details": "$details",
"count": "$count"
}
}
}
},
{
"$project": {
"_id": 0,
"event_date": "$_id",
"data": 1
}
}
])
In the above pipeline, the first step is the $unwind operator
{ "$unwind": "$pb_event" }
which comes in quite handy when the data is stored as an array. When the unwind operator is applied on a list data field, it will generate a new record for each and every element of the list data field on which unwind is applied. It basically flattens the data.
This is a necessary operation for the next pipeline stage, the $group step where you group the flattened documents by the deconstructed pb_event array fields event_date and event_type:
{
"$group": {
"_id": {
"event_date": "$pb_event.event_date",
"event_type": "$pb_event.event_type"
},
"details": {
"$push": {
"_id": "$_id",
"name": "$Name"
}
},
"count": { "$sum": 1 }
}
},
The $group pipeline operator is similar to the SQL's GROUP BY clause. In SQL, you can't use GROUP BY unless you use any of the aggregation functions. The same way, you have to use an aggregation function in MongoDB (called an accumulator operator) as well. You can read more about the aggregation functions here.
In this $group operation, the logic to calculate the count aggregate i.e. the total number of documents in the group using the $sum accumulator operator. Within the same pipeline, you can aggregate a list of the name and _id subdocuments by using the $push operator which returns an array of expression values for each group.
The preceding $group pipeline
{
"$group": {
"_id": "$_id.event_date",
"data": {
"$push": {
"event_type": "$_id.event_type",
"details": "$details",
"count": "$count"
}
}
}
}
will further aggregate the results from the last pipeline by grouping on the event_date, which forms basis of the desired output by creating a new data list using $push and then the final $project pipeline stage
{
"$project": {
"_id": 0,
"event_date": "$_id",
"data": 1
}
}
reshapes the documents fields by renaming the _id field to event_date and retaining the other field.

How to get mongodb deeply embeded document id

I have the following mongo document, which is part of a bigger document called attributes, which also has Colour and Size
> db.attributes.find({'name': {'en-UK': 'Fabric'}}).pretty()
{
"_id" : ObjectId("543261cda14c971132fa2b91"),
"values" : [
{
"source" : [
{
"_id" : ObjectId("543261cda14c971132fa2b79"),
"name" : {
"en-UK" : "Combed Cotton"
}
},
],
"name" : [
{
"_id" : ObjectId("543261cda14c971132fa2b85"),
"name" : {
"en-UK" : "Brushed 3-ply"
}
},
{
"_id" : ObjectId("543261cda14c971132fa2b8f"),
"name" : {
"en-UK" : "Plain Weave"
}
},
{
"_id" : ObjectId("543261cda14c971132fa2b90"),
"name" : {
"en-UK" : "1x1 Rib"
}
}
]
}
],
"name" : {
"en-UK" : "Fabric"
}
}
I am trying to return the _id for a sub document and have the following:
db.attributes.aggregate([
{ '$match': {'name.en-UK': 'Fabric'} },
{ '$unwind' : '$values' },
{ '$project': { 'name' : '$values.name'} },
{ '$match': { '$and': [{"name.name.en-UK" : "1x1 Rib"} ] }}
])
What is the correct way to do this?
Also, the values of Fabric is an array with two items, source and name, but if I populate it like:
> db.attributes.find({'name': {'en-UK': 'Fabric'}}).pretty()
{
"_id" : ObjectId("543261cda14c971132fa2b91"),
"values" : {
"source" : [{ ... }]
"name": [{ ... }]
}
}
I get the following error
"errmsg" : "exception: $unwind: value at end of field path must be an array"
But if I wrap it inside a square brackets this then works, so that
> db.attributes.find({'name': {'en-UK': 'Fabric'}}).pretty()
{
"_id" : ObjectId("543261cda14c971132fa2b91"),
"values" : [{
"source" : [{ ... }],
"name": [{ ... }]
}]
}
what am I missing as values is an array of two objects, source and name each containing a list of arrays
Any advice much appreciated
What you seem to be "missing" here is that "some" of your documents do either not contain a "value" property at all or at the very least it is "not an array". This is the basic context of the error you have been given.
Fortunately there are a couple of ways to get around this. Namely, either "testing" for the presence of an array when submitting you original query. Or actually "substituting" the missing element for some kind of array when processing the pipeline.
Here are both approaches in what is effectively an redundant form since the first $match condition really sorts this out:
db.attributes.aggregate([
{ "$match": {
"name.en-UK": "Fabric",
"values.0": { "$exists": true }
}},
{ "$project": {
"name": 1,
"values": { "$ifNull": [ "$values", [] ] }
}},
{ "$unwind": "$values" },
{ "$unwind": "$values.name" },
{ "$match": { "values.name.name.en-UK" : "1x1 Rib" }}
])
So as I said. Really redundant in that the initial $match actually asks if an "initial array element" actually exists. Which kind of means that there is an array there.
The second $project phase actually uses the $ifNull operator to "fill in" a value ( or basically an empty array ) where the tested element does not exist. We tested for that anyway before, but this demonstrates the different approaches.
But the basic idea id either "avoiding" or "filling-in" where your document does not have the expected data that you want to process. Which is the cause of your error.

Grouping records in nested documents

I have a document like this:
{
"_id" : ObjectId("533e6ab0ef2188940b00002c"),
"uin" : "1396599472869",
"vm" : {
"0" : {
"draw" : "01s",
"count" : "2",
"type" : "",
"data" : {
"title" : "K1"
},
"child" : [
"1407484608965"
]
},
"1407484608965" : {
"data" : {
"title" : "K2",
"draw" : "1407473540857",
"count" : "1",
"type" : "Block"
},
"child" : [
"1407484647012"
]
},
"1407484647012" : {
"data" : {
"title" : "K3",
"draw" : "03.8878.98",
"count" : "1",
"type" : "SB"
},
"child" : [
"1407484762473"
]
},
"1407484762473" : {
"data" : {
"type" : "SB",
"title" : "D1",
"draw" : "7984",
"count" : "1"
},
"child" : []
}
}
}
How to group all records with condition (type="Block")?
I've tried:
db.ITR.aggregate({$match:{"uin":"1396599472869"}},{$project:{"vm":1}},{$group:{_id:null,r1:{$push:"$vm"}}},{$unwind:"$r1"},{$group:{_id:null,r2:{$push:"$r1"}}},{$unwind:"$r2"})
But the result is still in the form of an object and not an array. With "MapReduce" I did not get.
Your problem here is basically with the way you currently have your document structured. The usage of "keys" under "vm" here that actually identify data points does not play well with the standard query forms and the aggregation framework in general.
It also is generally not a very good pattern, as in order to access any part under "vm" you need to specify the "exact path" to the data. So looking for type "Block" requires this:
db.collection.find({
"$or": [
{ "vm.0.type": "Block" },
{ "vm.1407484608965.type": "Block" }
{ ... }
]
})
And so on. You cannot "wildcard" field names like this so the exact path is required.
A better approach to modelling is to use an array instead, and move that inner key inside the documents:
{
"_id" : ObjectId("533e6ab0ef2188940b00002c"),
"uin" : "1396599472869",
"vm" : [
{
"key": 0,
"draw" : "01s",
"count" : "2",
"type" : "",
"data" : {
"title" : "K1"
},
"child" : [
"1407484608965"
]
},
{
"key": "1407484608965",
"title" : "K2",
"draw" : "1407473540857",
"count" : "1",
"type" : "Block",
"child" : [
"1407484647012"
]
},
{
"key": "1407484647012",
"title" : "K3",
"draw" : "03.8878.98",
"count" : "1",
"type" : "SB",
"child" : [
"1407484762473"
]
}
]
}
This allows you to query for documents that contain the matching property by a common path, which greatly simplifies things:
db.collection.find({ "vm.type": "Block" })
Or if you want to "filter" the array contents so that only those "sub-documents" that match are returned you can do this:
db.collection.aggregate([
{ "$match": { "vm.type": "Block" } },
{ "$unwind": "$vm" },
{ "$match": { "vm.type": "Block" } },
{ "$group": {
"_id": "$_id",
"uin": { "$first": "$uin" },
"vm": { "$push": "$vm" }
}}
])
Or even possibly this with MongoDB 2.6 or greater:
db.collection.aggregate([
{ "$match": { "vm.type": "Block" } },
{ "$project": {
"uin": 1,
"vm": {
"$setDifference": [
{ "$map": {
"input": "$vm",
"as": "el",
"in": {"$cond": [
{ "$eq": [ "$$el.type", "Block" ] },
"$$el",
false
]}
}},
[false]
]
}
}}
])
Or any other operation, which is simplified to traverse now the data is structured that way. But as your data presently stands your only option to "traverse keys" is to use JavaScript operations, which is much slower than being able to query in a proper way:
db.collection.find(function() {
return Object.keys(this.vm).some(function(x) {
return this.vm[x].type == "Block"
})
})
Or with similar object processing using mapReduce but essentially with no other way to access the fields with fixed paths that vary all the time.
Perhaps this was a design entered into to avoid having "nested arrays" which is where the "child" element would be placed. Of course this poses a problem with updates. But really if any element should not be an array it is probably the "inner" element such as "child", which could have some kind of structure that does not use an array.
So the key is to look at restructuring, as this will likely suit the patterns that you want without causing performance problems that JavaScript traversal will introduce.

Aggregation framework flatten subdocument data with parent document

I am building a dashboard that rotates between different webpages. I am wanting to pull all slides that are part of the "Test" deck and order them appropriately. After the query my result would ideally look like.
[
{ "url" : "http://10.0.1.187", "position": 1, "duartion": 10 },
{ "url" : "http://10.0.1.189", "position": 2, "duartion": 3 }
]
I currently have a dataset that looks like the following
{
"_id" : ObjectId("53a612043c24d08167b26f82"),
"url" : "http://10.0.1.189",
"decks" : [
{
"title" : "Test",
"position" : 2,
"duration" : 3
}
]
}
{
"_id" : ObjectId("53a6103e3c24d08167b26f81"),
"decks" : [
{
"title" : "Test",
"position" : 1,
"duration" : 2
},
{
"title" : "Other Deck",
"position" : 1,
"duration" : 10
}
],
"url" : "http://10.0.1.187"
}
My attempted query looks like:
db.slides.aggregate([
{
"$match": {
"decks.title": "Test"
}
},
{
"$sort": {
"decks.position": 1
}
},
{
"$project": {
"_id": 0,
"position": "$decks.position",
"duration": "$decks.duration",
"url": 1
}
}
]);
But it does not yield my desired results. How can I query my dataset and get my expected results in a optimal way?
Well to truly "flatten" the document as your title suggests then $unwind is always going to be employed as there really is not other way to do that. There are however some different approaches if you can live with the array being filtered down to the matching element.
Basically speaking, if you really only have one thing to match in the array then your fastest approach is to simply use .find() matching the required element and projecting:
db.slides.find(
{ "decks.title": "Test" },
{ "decks.$": 1 }
).sort({ "decks.position": 1 }).pretty()
That is still an array but as long as you have only one element that matches then this does work. Also the items are sorted as expected, though of course the "title" field is not dropped from the matched documents, as that is beyond the possibilities for simple projection.
{
"_id" : ObjectId("53a6103e3c24d08167b26f81"),
"decks" : [
{
"title" : "Test",
"position" : 1,
"duration" : 2
}
]
}
{
"_id" : ObjectId("53a612043c24d08167b26f82"),
"decks" : [
{
"title" : "Test",
"position" : 2,
"duration" : 3
}
]
}
Another approach, as long as you have MongoDB 2.6 or greater available, is using the $map operator and some others in order to both "filter" and re-shape the array "in-place" without actually applying $unwind:
db.slides.aggregate([
{ "$project": {
"url": 1,
"decks": {
"$setDifference": [
{
"$map": {
"input": "$decks",
"as": "el",
"in": {
"$cond": [
{ "$eq": [ "$$el.title", "Test" ] },
{
"position": "$$el.position",
"duration": "$$el.duration"
},
false
]
}
}
},
[false]
]
}
}},
{ "$sort": { "decks.position": 1 }}
])
The advantage there is that you can make the changes without "unwinding", which can reduce processing time with large arrays as you are not essentially creating new documents for every array member and then running a separate $match stage to "filter" or another $project to reshape.
{
"_id" : ObjectId("53a6103e3c24d08167b26f81"),
"decks" : [
{
"position" : 1,
"duration" : 2
}
],
"url" : "http://10.0.1.187"
}
{
"_id" : ObjectId("53a612043c24d08167b26f82"),
"url" : "http://10.0.1.189",
"decks" : [
{
"position" : 2,
"duration" : 3
}
]
}
You can again either live with the "filtered" array or if you want you can again "flatten" this truly by adding in an additional $unwind where you do not need to filter with $match as the result already contains only the matched items.
But generally speaking if you can live with it then just use .find() as it will be the fastest way. Otherwise what you are doing is fine for small data, or there is the other option for consideration.
Well as soon as I posted I realized I should be using an $unwind. Is this query the optimal way to do it, or can it be done differently?
db.slides.aggregate([
{
"$unwind": "$decks"
},
{
"$match": {
"decks.title": "Test"
}
},
{
"$sort": {
"decks.position": 1
}
},
{
"$project": {
"_id": 0,
"position": "$decks.position",
"duration": "$decks.duration",
"url": 1
}
}
]);