MongoDB Aggregation Projection - mongodb

If I have a collection as follows:
db.cafe.insert({name: "Cafe1", customers: [{name: "David", foods: [{name : "cheese"}, {name: "beef"}]}, {name: "Bill", foods: [{name: "fish"}]} ]})
db.cafe.find().pretty()
{
"_id" : ObjectId("54f5ae58baed23b7a34fccb6"),
"name" : "Cafe1",
"customers" : [
{
"name" : "David",
"foods" : [
{
"name" : "cheese"
},
{
"name" : "beef"
}
]
},
{
"name" : "Bill",
"foods" : [
{
"name" : "fish"
}
]
}
]
}
How can I extract an array containing just the food objects for people called "David".
Desired output is just the array of foods, i.e:
[{name: "cheese"}, {name: "beef"}]
I have tried an aggregation pipeline that unwinds the cafes customers, then matches on name then projects the food, e.g:
db.cafe.aggregate( [{$unwind : "$customers"}, {$match : {"customers.name": "David"}}, {$project : {"customers.foods": 1, _id : 0}
}] ).pretty()
{
"customers" : {
"foods" : [
{
"name" : "cheese"
},
{
"name" : "beef"
}
]
}
}
This seems close to the desired result, however, I'm left with the issue that the foods I want are referenced as an array under the property customers.foods. I would like the result to directly be:
[
{
"name" : "cheese"
},
{
"name" : "beef"
}
]
is there a way I can achieve the desired output?

You are doing your projection wrong.
db.cafe.aggregate( [
{ "$match" : { "customers.name": "David" }},
{ "$unwind" : "$customers" },
{ "$project" : { "foods": "$customers.foods", "_id": 0 }}
])
Output
{ "foods" : [ { "name" : "cheese" }, { "name" : "beef" } ] }

You can also get (something very, very close to) your desired output with a regular query:
> db.cafe.find({ "customers.name" : "David" }, { "customers.$.foods" : 1, "_id" : 0 })
{ "customers" : [ { "name" : "David", "foods" : [ { "name" : "cheese" }, { "name" : "beef" } ] } ] }
Customers will be an array containing just the first object with name : "David". You should prefer this approach to the aggregation as it's vastly more performant. You can extract the foods array in client code.

Related

MongoDB - Fetch an object from deep subdocuments

How to query an object from an Array inside an Array, and get it as a top-level object? For example, consider the following record.
{
"subjects": [
{
"name": "English",
"teachers": [
{
"name": "Mark" /* Trying to get this object*/
},
{
"name": "John"
}
]
}
]
}
I am trying to get the following object out as the top-level object.
{
"name": "Mark"
}
You need to use the aggregation framework to do exactly what you're asking for.
Here I entered the document you gave into collection: foo.
> db.foo.find().pretty()
{
"_id" : ObjectId("57ceed3d31484d5b491eaae9"),
"subjects" : [
{
"name" : "English",
"teachers" : [
{
"name" : "Mark"
},
{
"name" : "John"
}
]
}
]
}
Using $unwind to unravel our array we then enter our first stage of the aggregation pipeline:
> db.foo.aggregate([
... {$unwind: "$subjects"}
... ]).pretty()
{
"_id" : ObjectId("57ceed3d31484d5b491eaae9"),
"subjects" : {
"name" : "English",
"teachers" : [
{
"name" : "Mark"
},
{
"name" : "John"
}
]
}
}
Subjects was an array of length 1 so the only difference here is one less set of [] array brackets.
We need to unwind again.
> db.foo.aggregate([
... {$unwind: "$subjects"},
... {$unwind: "$subjects.teachers"}
... ]).pretty()
{
"_id" : ObjectId("57ceed3d31484d5b491eaae9"),
"subjects" : {
"name" : "English",
"teachers" : {
"name" : "Mark"
}
}
}
{
"_id" : ObjectId("57ceed3d31484d5b491eaae9"),
"subjects" : {
"name" : "English",
"teachers" : {
"name" : "John"
}
}
}
Now we turned our array of length '2' into two separate documents. The first one with subjects.teachers.name = Mark and the second with subjects.teachers.name = John.
We only want to return the case where name = Mark so we need to add a $match stage to our pipeline.
> db.foo.aggregate([
... {$unwind: "$subjects"},
... {$unwind: "$subjects.teachers"},
... {$match: {"subjects.teachers.name": "Mark"}}
... ]).pretty()
{
"_id" : ObjectId("57ceed3d31484d5b491eaae9"),
"subjects" : {
"name" : "English",
"teachers" : {
"name" : "Mark"
}
}
}
Ok! Now we are only matching on the case where name: Mark.
Let's add a $project case to shape our input how we want.
> db.foo.aggregate([
... {$unwind: "$subjects"},
... {$unwind: "$subjects.teachers"},
... {$match: {"subjects.teachers.name": "Mark"}},
... {$project: {"name": "$subjects.teachers.name", "_id": 0}}
... ]).pretty()
{ "name" : "Mark" }

MongoDB filtering out subdocuments with lookup aggregation

Our project database has a capped collection called values which gets updated every few minutes with new data from sensors. These sensors all belong to a single sensor node, and I would like to query the last data from these nodes in a single aggregation. The problem I am having is filtering out just the last of ALL the types of sensors while still having only one (efficient) query. I looked around and found the $group argument, but I can't seem to figure out how to use it correctly in this case.
The database is structured as follows:
nodes:
{
"_id": 681
"sensors": [
{
"type": "foo"
},
{
"type": "bar"
}
]
}
values:
{
"_id" : ObjectId("570cc8b6ac55850d5740784e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"type" : "foo",
"nodeid" : 681,
"value" : 10
}
{
"_id" : ObjectId("190ac8b6ac55850d5740776e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"type" : "bar",
"nodeid" : 681,
"value" : 20
}
{
"_id" : ObjectId("167bc997bb66750d5740665e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"type" : "bar",
"nodeid" : 200,
"value" : 20
}
{
"_id" : ObjectId("110cc9c6ac55850d5740784e"),
"timestamp" : ISODate("2016-04-09T12:06:46.344Z"),
"type" : "foo",
"nodeid" : 681,
"value" : 12
}
so let's imagine I want the data from node 681, I would want a structure like this:
nodes:
{
"_id": 681
"sensors": [
{
"_id" : ObjectId("570cc8b6ac55850d5740784e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"type" : "foo",
"nodeid" : 681,
"value" : 10
},
{
"_id" : ObjectId("190ac8b6ac55850d5740776e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"type" : "bar",
"nodeid" : 681,
"value" : 20
}
]
}
Notice how one value of foo is not queried, because I want to only get the latest value possible if there are more than one value (which is always going to be the case). The ordering of the collection is already according to the timestamp because the collection is capped.
I have this query, but it just gets all the values from the database (which is waaay too much to do in a lifetime, let alone one request of the web app), so I was wondering how I would filter it before it gets aggregated.
query:
db.nodes.aggregate(
[
{
$unwind: "$sensors"
},
{
$match:{
nodeid: 681
}
},
{
$lookup:{
from: "values", localField: "sensors.type", foreignField: "type", as: "sensors"
}
}
}
]
)
Try this
// Pipeline
[
// Stage 1 - sort the data collection if not already done (optional)
{
$sort: {
"timestamp":1
}
},
// Stage 2 - group by type & nodeid then get first item found in each group
{
$group: {
"_id":{type:"$type",nodeid:"$nodeid"},
"sensors": {"$first":"$$CURRENT"} //consider using $last if your collection is on reverse
}
},
// Stage 3 - project the fields in desired
{
$project: {
"_id":"$sensors._id",
"timestamp":"$sensors.timestamp",
"type":"$sensors.type",
"nodeid":"$sensors.nodeid",
"value":"$sensors.value"
}
},
// Stage 4 - group and push it to array sensors
{
$group: {
"_id":{nodeid:"$nodeid"},
"sensors": {"$addToSet":"$$CURRENT"}
}
}
]
as far as I got document structure, there is no need to use $lookup as all data is in readings(values) collection.
Please see proposed solution:
db.readings.aggregate([{
$match : {
nodeid : 681
}
},
{
$group : {
_id : {
type : "$type",
nodeid : "$nodeid"
},
readings : {
$push : {
timestamp : "$timestamp",
value : "$value",
id : "$_id"
}
}
}
}, {
$project : {
_id : "$_id",
readings : {
$slice : ["$readings", -1]
}
}
}, {
$unwind : "$readings"
}, {
$project : {
_id : "$readings.id",
type : "$_id.type",
nodeid : "$_id.nodeid",
timestamp : "$readings.timestamp",
value : "$readings.value",
}
}, {
$group : {
_id : "$nodeid",
sensors : {
$push : {
_id : "$_id",
timestamp : "$timestamp",
value : "$value",
type:"$type"
}
}
}
}
])
and output:
{
"_id" : 681,
"sensors" : [
{
"_id" : ObjectId("110cc9c6ac55850d5740784e"),
"timestamp" : ISODate("2016-04-09T12:06:46.344Z"),
"value" : 12,
"type" : "foo"
},
{
"_id" : ObjectId("190ac8b6ac55850d5740776e"),
"timestamp" : ISODate("2016-04-12T12:06:46.344Z"),
"value" : 20,
"type" : "bar"
}
]
}
Any comments welcome!

Sorting objects in array within mongodb

I've seen this question all over google/SO/mongo docs, and I've tried to implement the solution, but it's not working for me. I have the following test database:
> db.test.find().pretty()
{
"_id" : ObjectId("56b4ab167db9acd913ce6e07"),
"state" : "HelloWorld",
"items" : [
{
"guid" : "123"
},
{
"guid" : "124"
},
{
"guid" : "123"
}
]
}
And I want to sort by the "guid" element of items. Running the sort commands yields:
> db.test.find().sort( {"items.guid" : 1}).pretty()
{
"_id" : ObjectId("56b4ab167db9acd913ce6e07"),
"state" : "HelloWorld",
"items" : [
{
"guid" : "123"
},
{
"guid" : "124"
},
{
"guid" : "123"
}
]
}
How can I sort by the "guid" element, so that the returned output of "items" is the 123, 123, and 124 guids (essentially move the child elements of "items" so that they're sorted by "guid")?
EDIT: I've also tried to use the $orderby command, doesn't accomplish what I want:
> db.test.find({ $query : {}, $orderby: {'items.guid' : 1} }).pretty()
{
"_id" : ObjectId("56b4ab167db9acd913ce6e07"),
"state" : "HelloWorld",
"items" : [
{
"guid" : "123"
},
{
"guid" : "124"
},
{
"guid" : "123"
}
]
}
Here is how it can be done using aggregate
db.test.aggregate([
{
$unwind : '$items'
},
{
$sort : {'items.guid' : 1}
},
{
$group : {
_id : '$_id',
state : {$first : '$state'},
items : {
$push : {'guid' : '$items.guid'}
}
}
}
]).pretty()
This is the output from this command.
{
"_id" : ObjectId("56b4ab167db9acd913ce6e07"),
"state" : "HelloWorld",
"items" : [
{
"guid" : "123"
},
{
"guid" : "123"
},
{
"guid" : "124"
}
]
}

Find duplicate key in embedded sub document in mongodb

I am trying to craft a query that will allow me to find duplicate keys in subdocument in MongoDB.
It needs to be able to query any number of documents and see what keys are duplicated across them in a subdocument. The key of my subdocument is called attributes and I need to be able to target a particular query of documents and pull out duplicate attribute keys that they all share.
EDIT:
I forgot to mention that I do not know the names of the attributes ahead of time. I need to be able to essentially select distinct attributes that they share and aggregate the values.
Collection Sample:
[
{
sku: '123',
attributes: {
size: 'L',
custom: 7
}
},
{
sku: '456',
attributes: {
size: 'M'
}
},
{
sku: 'abc',
attributes: {
material: 'cotton'
size: 'S'
}
}
]
Desired Result (if possible):
{
size: [' S', 'M', 'L']
}
If the desired result is not possible I would at least like to be able to get back [ 'size' ]
This process needs to be optimized as much as possible and I just cant seem to get a query just right to return what I need, any help is greatly appreciated =)
Here is what I have so far
db.getCollection('myCollection').aggregate([
{ $match: {
_id: { $in: [ObjectId("55158b0bd6076278295cf022"), ObjectId("55158b0bd6076278295cf021"), ObjectId("55158b0bd6076278295cf01f") ] }
}
},
{ $project: { attributes: 1 }},
{ $group: { _id: '$attributes' } }
])
Which products this output:
{
"result" : [
{
"_id" : {
"shirt_size" : "S",
"shirt_color" : "Blue",
"custom_attr" : "adsfasdf"
}
},
{
"_id" : {
"shirt_size" : "M",
"shirt_color" : "Green"
}
},
{
"_id" : {
"shirt_size" : "L",
"shirt_color" : "Red"
}
}
],
"ok" : 1.0000000000000000,
"$gleStats" : {
"lastOpTime" : Timestamp(1427475045, 1),
"electionId" : ObjectId("54f7c1edf8e5ff44cec194b6")
}
}
I feel like it is close and I am just missing the last step :(
I think you need to $unwind the array, and then $group it and use $sum to count the appearance, then everything with sum > 1 is a duplicate.
Links:
http://docs.mongodb.org/manual/reference/operator/aggregation/unwind/
http://docs.mongodb.org/manual/reference/operator/aggregation/group/
http://docs.mongodb.org/manual/reference/operator/aggregation/sum/
The $addToSet(aggregation) returns an array of unique values - http://docs.mongodb.org/manual/reference/operator/aggregation/addToSet/
Using the following aggregation (get unique sizes per Doc):
db.coll1.aggregate([
{$unwind : "$testdoc"},
{$group : {_id: "$_id", size: {$addToSet: "$testdoc.attributes.size"}}}
])
Gives the following result:
{
"result" : [
{
"_id" : ObjectId("551621fe6155a7741a0d328a"),
"size" : [
"M",
"L"
]
},
{
"_id" : ObjectId("551621fe6155a7741a0d328b"),
"size" : [
"L"
]
},
{
"_id" : ObjectId("551621fe6155a7741a0d3289"),
"size" : [
"S",
"M",
"L"
]
}
],
"ok" : 1
}
The following aggregation returns unique sizes across all docs:
db.coll1.aggregate([
{$unwind : "$testdoc"},
{$group :
{_id: "AllSizes", size: {$addToSet: "$testdoc.attributes.size"}}} ])
Result:
{
"result" : [
{
"_id" : "AllSizes",
"size" : [
"S",
"M",
"L"
]
}
],
"ok" : 1
}
Based on the following Docs:
> db.coll1.find().pretty()
{
"_id" : ObjectId("551621fe6155a7741a0d3289"),
"testdoc" : [
{
"sku" : "123",
"attributes" : {
"size" : "L",
"custom" : 7
}
},
{
"sku" : "456",
"attributes" : {
"size" : "M"
}
},
{
"sku" : "abc",
"attributes" : {
"material" : "cotton",
"size" : "S"
}
}
]
}
{
"_id" : ObjectId("551621fe6155a7741a0d328a"),
"testdoc" : [
{
"sku" : "123",
"attributes" : {
"size" : "L",
"custom" : 7
}
},
{
"sku" : "456",
"attributes" : {
"size" : "M"
}
},
{
"sku" : "abc",
"attributes" : {
"material" : "cotton",
"size" : "M"
}
}
]
}
{
"_id" : ObjectId("551621fe6155a7741a0d328b"),
"testdoc" : [
{
"sku" : "123",
"attributes" : {
"size" : "L",
"custom" : 7
}
},
{
"sku" : "456",
"attributes" : {
"size" : "L"
}
},
{
"sku" : "abc",
"attributes" : {
"material" : "cotton",
"size" : "L"
}
}
]
}

Aggregate of different subtypes in document of a collection

abstract document in collection md given:
{
vals : [{
uid : string,
val : string|array
}]
}
the following, partially correct aggregation is given:
db.md.aggregate(
{ $unwind : "$vals" },
{ $match : { "vals.uid" : { $in : ["x", "y"] } } },
{
$group : {
_id : { uid : "$vals.uid" },
vals : { $addToSet : "$vals.val" }
}
}
);
that may lead to the following result:
"result" : [
{
"_id" : {
"uid" : "x"
},
"vals" : [
[
"24ad52bc-c414-4349-8f3a-24fd5520428e",
"e29dec2f-57d2-43dc-818a-1a6a9ec1cc64"
],
[
"5879b7a4-b564-433e-9a3e-49998dd60b67",
"24ad52bc-c414-4349-8f3a-24fd5520428e"
]
]
},
{
"_id" : {
"uid" : "y"
},
"vals" : [
"0da5fcaa-8d7e-428b-8a84-77c375acea2b",
"1721cc92-c4ee-4a19-9b2f-8247aa53cfe1",
"5ac71a9e-70bd-49d7-a596-d317b17e4491"
]
}
]
as x is the result aggregated on documents containing an array rather than a string, the vals in the result is an array of arrays. what i look for in this case is to have a flattened array (like the result for y).
for me it seems like that what i want to achieve by one aggegration call only, is currently not supported by any given operation as e.g. a type conversion cannot be done or unwind expectes in every case an array as input type.
is map reduce the only option i have? if not ... any hints?
thanks!
You can use the aggregation to do the computation you want without changing your schema (though you might consider changing your schema simply to make queries and aggregations of this field easier to write).
I broke up the pipeline into multiple steps for readability. I also simplified your document slightly, again for readability.
Sample input:
> db.md.find().pretty()
{
"_id" : ObjectId("512f65c6a31a92aae2a214a3"),
"uid" : "x",
"val" : "string"
}
{
"_id" : ObjectId("512f65c6a31a92aae2a214a4"),
"uid" : "x",
"val" : "string"
}
{
"_id" : ObjectId("512f65c6a31a92aae2a214a5"),
"uid" : "y",
"val" : "string2"
}
{
"_id" : ObjectId("512f65e8a31a92aae2a214a6"),
"uid" : "y",
"val" : [
"string3",
"string4"
]
}
{
"_id" : ObjectId("512f65e8a31a92aae2a214a7"),
"uid" : "z",
"val" : [
"string"
]
}
{
"_id" : ObjectId("512f65e8a31a92aae2a214a8"),
"uid" : "y",
"val" : [
"string1",
"string2"
]
}
Pipeline stages:
> project1 = {
"$project" : {
"uid" : 1,
"val" : 1,
"isArray" : {
"$cond" : [
{
"$eq" : [
"$val.0",
[ ]
]
},
true,
false
]
}
}
}
> project2 = {
"$project" : {
"uid" : 1,
"valA" : {
"$cond" : [
"$isArray",
"$val",
[
null
]
]
},
"valS" : {
"$cond" : [
"$isArray",
null,
"$val"
]
},
"isArray" : 1
}
}
> unwind = { "$unwind" : "$valA" }
> project3 = {
"$project" : {
"_id" : 0,
"uid" : 1,
"val" : {
"$cond" : [
"$isArray",
"$valA",
"$valS"
]
}
}
}
Final aggregation:
> db.md.aggregate(project1, project2, unwind, project3, group)
{
"result" : [
{
"_id" : "z",
"vals" : [
"string"
]
},
{
"_id" : "y",
"vals" : [
"string1",
"string4",
"string3",
"string2"
]
},
{
"_id" : "x",
"vals" : [
"string"
]
}
],
"ok" : 1
}
If you modify your schema using always "vals.val" field as an array field (even when the record contains only one element) you can do it easily as follows:
db.test_col.insert({
vals : [
{
uid : "uuid1",
val : ["value1"]
},
{
uid : "uuid2",
val : ["value2", "value3"]
}]
});
db.test_col.insert(
{
vals : [{
uid : "uuid2",
val : ["value4", "value5"]
}]
});
Using this approach you only need to use two $unwind operations: one unwinds the "parent" array and the second unwinds every "vals.val" value. So, querying like
db.test_col.aggregate(
{ $unwind : "$vals" },
{ $unwind : "$vals.val" },
{
$group : {
_id : { uid : "$vals.uid" },
vals : { $addToSet : "$vals.val" }
}
}
);
You can obtain your expected value:
{
"result" : [
{
"_id" : {
"uid" : "uuid2"
},
"vals" : [
"value5",
"value4",
"value3",
"value2"
]
},
{
"_id" : {
"uid" : "uuid1"
},
"vals" : [
"value1"
]
}
],
"ok" : 1
}
And no, you can't execute this query using your current schema, since $unwind fails when the field isn't an array field.