Related
I want to set a variable at a function level from inside the aggregate map operator. Here is a snippet of what I have till now.
$project: {
collectionId: 1,
name: 1,
description: 1,
image: 1,
created: 1,
updated: 1,
isPresent: "no",
"products": {
$map: {
"input": "$products",
as: "product",
in: {
"title": "$$product.name",
"imageUrl": "$$product.mainImage",
"productId": "$$product.productId",
"$isPresent": "yes"
}
}
},
total: {
$size: "$products"
},
userId: "$user.userId",
userName: "$user.name",
}
I want to set the isPresent node value to yes inside the map operator at in based on a condition. For the time being I have kept it to just yes. But this doesn't work.
What is the way to do it?
You can use $in and $cond operators :
db.collection.aggregate([
/** `$products.productId` gives an array of 'productId''s from products array */
{
$addFields: { isPresent: { $cond: [ { $in: [ 2, "$products.productId" ] }, "Yes", "No" ] } }
}
])
Test : mongoplayground
Note :
As we're using $in each document going through this has to have products array, otherwise $in will error out, if there are any such cases you need to have to condition to skip for those docs which don't have products array.
I have a MongoDB collection called "ballots" and it looks like this:
{
"_id" : "msw9ofwQSj58qiPvY",
"ballotName" : "This is the ballot name",
"votes" : [
{ "unitNumber" : "30", "voteResult" : "No", "voteDate" : ISODate("2015-02-24T05:03:22.937Z") },
{ "unitNumber" : "30", "voteResult" : "Yes", "voteDate" : ISODate("2015-02-24T05:20:02.479Z") }
]
}
I want to see only the LATEST "votes.voteResult" for this _id and by unitNumber:30. I try:
db.ballots.find(
{ $and: [
{_id: 'msw9ofwQSj58qiPvY'},
{'votes.unitNumber': '30'}
] },
{'votes.voteResult': 1, 'votes.voteDate': 1, _id: 0} )
Results:
{
"votes" : [
{
"voteResult": "No",
"voteDate" : ISODate("2015-02-24T05:03:22.937Z")
},
{
"voteResult": "Yes",
"voteDate" : ISODate("2015-02-24T05:20:02.479Z")
}
]
}
Looks good, however I only want ONE "voteResult" to return. I only want the latest one. I cannot seem to figure out how to get one of the nested result sets, AND ensure it is the most recent one. If it's any help, I am using Meteor/javascript which perhaps can handle the coding logic of this, but hoping a simple MongoDB query can just give me the result.
Any help is greatly appreciated. Thanks.
You can use sort and limit for these things:
db.ballots.find(
{ $and: [
{_id: 'msw9ofwQSj58qiPvY'},
{'votes.unitNumber': '30'},
] },
{'votes.voteResult': 1, 'votes.voteDate': 1, _id: 0}
).sort({'votes.voteDate':-1).limit(1);
You get both back, but if you sort them by voteDate and return the first one, you should have the one you want.
You might also be able to do it all server-side using aggregate instead of find:
db.ballots.aggregate(
{$match :
{$and: [
{_id: 'msw9ofwQSj58qiPvY'},
{'votes.unitNumber': '30'},
]}
},
{$unwind: "$votes"},
{$sort: {'votes.voteDate': -1}},
{$limit: 1},
{$project: {'votes.voteResult': 1, 'votes.voteDate': 1, _id: 0}}
);
I have next schema of mongo document:
{
"array1" : [
{
"array2" : [
{
"id" : 1,
"field" : 2
}
]
}
]
}
I want to find elements in array2 with {id: 1} and update its "field". I tried next:
db.filmCitation.update({ "array1.array2": {$elemMatch: {"id": 1, field: {$gte: 2}} } }, {$set: {"array1.array2.field": 23})
But I must specify right index in arrays. For one array can be used "$". And what can I do when needs to find in several arrays?
Thanks!
I know the theory of MongoDB and the fact that is doesn't support joins, and that I should use embeded documents or denormalize as much as possible, but here goes:
I have multiple documents, such as:
Users, which embed Suburbs, but also has: first name, last name
Suburbs, which embed States
Child, which embeds School, belongs to a User, but also has: first name, last name
Example:
Users:
{ _id: 1, first_name: 'Bill', last_name: 'Gates', suburb: 1 }
{ _id: 2, first_name: 'Steve', last_name: 'Jobs', suburb: 3 }
Suburb:
{ _id: 1, name: 'Suburb A', state: 1 }
{ _id: 2, name: 'Suburb B', state: 1 }
{ _id: 3, name: 'Suburb C', state: 3 }
State:
{ _id: 1, name: 'LA' }
{ _id: 3, name: 'NY' }
Child:
{ _id: 1, _user_id: 1, first_name: 'Little Billy', last_name: 'Gates' }
{ _id: 2, _user_id: 2, first_name: 'Little Stevie', last_name: 'Jobs' }
The search I need to implement is on:
first name, last name of Users and Child
State from Users
I know that I have to do multiple queries to get it done, but how can that be achieved? With mapReduce or aggregate?
Can you point out a solution please?
I've tried to use mapReduce but that didn't get me to have documents from Users which contained a state_id, so that's why I brought it up here.
This answer is outdated. Since version 3.2, MongoDB has limited support for left outer joins with the $lookup aggregation operator
MongoDB does not do queries which span multiple collections - period. When you need to join data from multiple collections, you have to do it on the application level by doing multiple queries.
Query collection A
Get the secondary keys from the result and put them into an array
Query collection B passing that array as the value of the $in-operator
Join the results of both queries programmatically on the application layer
Having to do this should be rather the exception than the norm. When you frequently need to emulate JOINs like that, it either means that you are still thinking too relational when you design your database schema or that your data is simply not suited for the document-based storage concept of MongoDB.
So now join is possible in mongodb and you can achieve this using $lookup and $facet aggregation here and which is probably the best way to find in multiple collections
db.collection.aggregate([
{ "$limit": 1 },
{ "$facet": {
"c1": [
{ "$lookup": {
"from": Users.collection.name,
"pipeline": [
{ "$match": { "first_name": "your_search_data" } }
],
"as": "collection1"
}}
],
"c2": [
{ "$lookup": {
"from": State.collection.name,
"pipeline": [
{ "$match": { "name": "your_search_data" } }
],
"as": "collection2"
}}
],
"c3": [
{ "$lookup": {
"from": State.collection.name,
"pipeline": [
{ "$match": { "name": "your_search_data" } }
],
"as": "collection3"
}}
]
}},
{ "$project": {
"data": {
"$concatArrays": [ "$c1", "$c2", "$c3" ]
}
}},
{ "$unwind": "$data" },
{ "$replaceRoot": { "newRoot": "$data" } }
])
You'll find MongoDB easier to understand if you take a denormalized approach to schema design. That is, you want to structure your documents the way the requesting client application understands them. Essentially, you are modeling your documents as domain objects with which the applicaiton deals. Joins become less important when you model your data this way. Consider how I've denormalized your data into a single collection:
{
_id: 1,
first_name: 'Bill',
last_name: 'Gates',
suburb: 'Suburb A',
state: 'LA',
child : [ 3 ]
}
{
_id: 2,
first_name: 'Steve',
last_name: 'Jobs',
suburb: 'Suburb C',
state 'NY',
child: [ 4 ]
}
{
_id: 3,
first_name: 'Little Billy',
last_name: 'Gates',
suburb: 'Suburb A',
state: 'LA',
parent : [ 1 ]
}
{
_id: 4,
first_name: 'Little Stevie',
last_name: 'Jobs'
suburb: 'Suburb C',
state 'NY',
parent: [ 2 ]
}
The first advantage is that this schema is far easier to query. Plus, updates to address fields are now consistent with the individual Person entity since the fields are embedded in a single document. Notice also the bidirectional relationship between parent and children? This makes this collection more than just a collection of individual people. The parent-child relationships mean this collection is also a social graph. Here are some resoures which may be helpful to you when thinking about schema design in MongoDB.
Here's a JavaScript function that will return an array of all records matching specified criteria, searching across all collections in the current database:
function searchAll(query,fields,sort) {
var all = db.getCollectionNames();
var results = [];
for (var i in all) {
var coll = all[i];
if (coll == "system.indexes") continue;
db[coll].find(query,fields).sort(sort).forEach(
function (rec) {results.push(rec);} );
}
return results;
}
From the Mongo shell, you can copy/paste the function in, then call it like so:
> var recs = searchAll( {filename: {$regex:'.pdf$'} }, {moddate:1,filename:1,_id:0}, {filename:1} )
> recs
Based on #brian-moquin and others, I made a set of functions to search entire collections with entire keys(fields) by simple keyword.
It's in my gist; https://gist.github.com/fkiller/005dc8a07eaa3321110b3e5753dda71b
For more detail, I first made a function to gather all keys.
function keys(collectionName) {
mr = db.runCommand({
'mapreduce': collectionName,
'map': function () {
for (var key in this) { emit(key, null); }
},
'reduce': function (key, stuff) { return null; },
'out': 'my_collection' + '_keys'
});
return db[mr.result].distinct('_id');
}
Then one more to generate $or query from keys array.
function createOR(fieldNames, keyword) {
var query = [];
fieldNames.forEach(function (item) {
var temp = {};
temp[item] = { $regex: '.*' + keyword + '.*' };
query.push(temp);
});
if (query.length == 0) return false;
return { $or: query };
}
Below is a function to search a single collection.
function findany(collection, keyword) {
var query = createOR(keys(collection.getName()));
if (query) {
return collection.findOne(query, keyword);
} else {
return false;
}
}
And, finally a search function for every collections.
function searchAll(keyword) {
var all = db.getCollectionNames();
var results = [];
all.forEach(function (collectionName) {
print(collectionName);
if (db[collectionName]) results.push(findany(db[collectionName], keyword));
});
return results;
}
You can simply load all functions in Mongo console, and execute searchAll('any keyword')
You can achieve this using $mergeObjects by MongoDB Driver
Example
Create a collection orders with the following documents:
db.orders.insert([
{ "_id" : 1, "item" : "abc", "price" : 12, "ordered" : 2 },
{ "_id" : 2, "item" : "jkl", "price" : 20, "ordered" : 1 }
])
Create another collection items with the following documents:
db.items.insert([
{ "_id" : 1, "item" : "abc", description: "product 1", "instock" : 120 },
{ "_id" : 2, "item" : "def", description: "product 2", "instock" : 80 },
{ "_id" : 3, "item" : "jkl", description: "product 3", "instock" : 60 }
])
The following operation first uses the $lookup stage to join the two collections by the item fields and then uses $mergeObjects in the $replaceRoot to merge the joined documents from items and orders:
db.orders.aggregate([
{
$lookup: {
from: "items",
localField: "item", // field in the orders collection
foreignField: "item", // field in the items collection
as: "fromItems"
}
},
{
$replaceRoot: { newRoot: { $mergeObjects: [ { $arrayElemAt: [ "$fromItems", 0 ] }, "$$ROOT" ] } }
},
{ $project: { fromItems: 0 } }
])
The operation returns the following documents:
{ "_id" : 1, "item" : "abc", "description" : "product 1", "instock" : 120, "price" : 12, "ordered" : 2 }
{ "_id" : 2, "item" : "jkl", "description" : "product 3", "instock" : 60, "price" : 20, "ordered" : 1 }
This Technique merge Object and return the result
Minime solution worked except that it required a fix:
var query = createOR(keys(collection.getName()));
need to add keyword as 2nd parameter to createOR call here.
Is it possible to group-by field name? Or do I need a different structure so I can group-by value?
I know we can use group by on values and we can unwind arrays, but is it possible to get total apples, pears and oranges owned by John amongst the three houses here without specifying "apples", "pears" and "oranges" explicitly as part of the query? (so NOT like this);
// total all the fruit John has at each house
db.houses.aggregate([
{
$group: {
_id: null,
"apples": { $sum: "$people.John.items.apples" },
"pears": { $sum: "$people.John.items.pears" },
"oranges": { $sum: "$people.John.items.oranges" },
}
},
])
In other words, can I group-by the first field-name under "items" and get the aggregate sum of apples:104, pears:202 and oranges:306, but also bananas, melons and anything else that might be there? Or do I need to restructure the data into an array of key/value pairs like categories?
db.createCollection("houses");
db.houses.remove();
db.houses.insert(
[
{
House: "birmingham",
categories : [
{
k : "location",
v : { d : "central" }
}
],
people: {
John: {
items: {
apples: 2,
pears: 1,
oranges: 3,
}
},
Dave: {
items: {
apples: 30,
pears: 20,
oranges: 10,
},
},
},
},
{
House: "London", categories: [{ k: "location", v: { d: "central" } }, { k: "type", v: { d: "rented" } }],
people: {
John: { items: { apples: 2, pears: 1, oranges: 3, } },
Dave: { items: { apples: 30, pears: 20, oranges: 10, }, },
},
},
{
House: "Cambridge", categories: [{ k: "type", v: { d: "rented" } }],
people: {
John: { items: { apples: 100, pears: 200, oranges: 300, } },
Dave: { items: { apples: 0.3, pears: 0.2, oranges: 0.1, }, },
},
},
]
);
Secondly, and more importantly, could I then also group by "house.categories.k" ? In other words, is it possible to find out how many "apples" "John" has in "rented" vs "owned" or "friends" houses (so group by "categories.k.type")?
Finally - if this is even possible, is it sensible? At first I thought it was quite useful to create dictionaries of nested objects using actual field names of the object, as it seemed a logical use of a document database, and it seemed to make the MR queries easier to write vs arrays, but now I'm starting to wonder if this is all a bad idea and having variable field names makes it very tricky/inefficient to write aggregation queries.
OK, so I think I have this partially solved. At least for the shape of data in the initial question.
// How many of each type of fruit does John have at each location
db.houses.aggregate([
{
$unwind: "$categories"
},
{
$match: { "categories.k": "location" }
},
{
$group: {
_id: "$categories.v.d",
"numberOf": { $sum: 1 },
"Total Apples": { $sum: "$people.John.items.apples" },
"Total Pears": { $sum: "$people.John.items.pears" },
}
},
])
which yields;
{
"result" : [
{
"_id" : "central",
"numberOf" : 2,
"Total Apples" : 4,
"Total Pears" : 2
}
],
"ok" : 1
}
Note that there's only "central", but if I had other "location"s in my DB I'd get a range of totals for each location. I wouldn't need the $unwind step if I had named properties instead of an array for "categories", but this is where I find the structure is at odds with itself. There are several keywords likely under "categories". The sample data shows "type" and "location" but there could be around 10 of these categorizations all with different values. So if I used named fields;
"categories": {
location: "london",
type: "owned",
}
...the problem I then have is indexing. I can't afford to simply index "location" since those are user-defined categories, and if 10,000 users choose 10,000 different ways of categorizing their houses I'd need 10,000 indexes, one for each field. But by making it an array I only need one on the array field itself. The downside is the $unwind step. I ran into this before with MapReduce. The last thing you want to be doing is a ForEach loop in JavaScript to cycle an array if you can help it. What you really want is to filter out the fields by name because it's much quicker.
Now this is all well and good where I already know what fruit I'm looking for, but if I don't, it's much harder. I can't (as far as I can see) $unwind or otherwise ForEach "people.John.items" here. If I could, I'd be overjoyed. So since the names of fruit are again user-defined, it looks like I need to convert them to an array as well, like this;
{
"people" : {
"John" : {
"items" : [
{ k:"apples", v:100 },
{ k:"pears", v:200 },
{ k:"oranges", v:300 },
]
},
}
}
So that now allows me get the fruit (where I don't know which fruit to look for) totalled, again by location;
db.houses.aggregate([
{
$unwind: "$categories"
},
{
$match: { "categories.k": "location" }
},
{
$unwind: "$people.John.items"
},
{
$group: { // compound key - thanks to Jenna
_id: { fruit:"$people.John.items.k", location:"$categories.v.v" },
"numberOf": { $sum: 1 },
"Total Fruit": { $sum: "$people.John.items.v" },
}
},
])
So now I'm doing TWO $unwinds. If you're thinking that looks grotesquely ineffecient, you'd be right. If I have just 10,000 house records, with 10 categories each, and 10 types of fruit, this query takes half a minute to run.
OK, so I can see that moving the $match before the $unwind improves things significantly but then it's the wrong output. I don't want an entry for every category, I want to filter out just the "location" categories.
I would have made this comment, but it's easier to format in a response text box.
{ _id: 1,
house: "New York",
people: {
John: {
items: {apples: 1, oranges:2}
}
Dave: {
items: {apples: 2, oranges: 1}
}
}
}
{ _id: 2,
house: "London",
people: {
John: {
items: {apples: 3, oranges:2}
}
Dave: {
items: {apples: 1, oranges:3}
}
}
}
Just to make sure I understand your question, is this what you're trying to accomplish?
{location: "New York", johnFruit:3}
{location: "London", johnFruit: 5}
Since categories is not nested under house, you can't group by "house.categories.k", but you can use a compound key for the _id of $group to get this result:
{ $group: _id: {house: "$House", category: "$categories.k"}
Although "k" doesn't contain the information that you're presumably trying to group by. And as for "categories.k.type", type is the value of k, so you can't use this syntax. You would have to group by "categories.v.d".
It may be possible with your current schema to accomplish this aggregation using $unwind, $project, possibly $match, and finally $group, but the command won't be pretty. If possible, I would highly recommend restructuring your data to make this aggregation much simpler. If you would like some help with schema, please let us know.
I'm not sure if this is a possible solution, but what if you begin the aggregation process by determining the number of different locations using distinct(), and run separate aggregation commands for each location? distinct() may not be efficient, but every subsequent aggregation will be able to use $match, and therefore, the index on categories. You could use the same logic to count the fruit for "categories.type".
{
"_id" : 1,
"house" : "New York",
"people" : {
"John" : [{"k" : "apples","v" : 1},{"k" : "oranges","v" : 2}],
"Dave" : [{"k" : "apples","v" : 2},{"k" : "oranges","v" : 1}]
},
"categories" : [{"location" : "central"},{"type" : "rented"}]
}
{
"_id" : 2,
"house" : "London",
"people" : {
"John" : [{"k" : "apples","v" : 3},{"k" : "oranges","v" : 2}],
"Dave" : [{"k" : "apples","v" : 3},{"k" : "oranges","v" : 1}]
},
"categories" : [{"location" : "suburb"},{"type" : "rented"}]
}
{
"_id" : 3,
"house" : "London",
"people" : {
"John" : [{"k" : "apples","v" : 0},{"k" : "oranges","v" : 1}],
"Dave" : [{"k" : "apples","v" : 2},{"k" : "oranges","v" : 4}]
},
"categories" : [{"location" : "central"},{"type" : "rented"}]
}
Run distinct() and iterate through the results by running aggregate() commands for each unique value of "categories.location":
db.agg.distinct("categories.location")
[ "central", "suburb" ]
db.agg.aggregate(
{$match: {categories: {location:"central"}}}, //the index entry is on the entire
{$unwind: "$people.John"}, //document {location:"central"}, so
{$group:{ //use this syntax to use the index
_id:"$people.John.k",
"numberOf": { $sum: 1 },
"Total Fruit": { $sum: "$people.John.v"}
}
}
)
{
"result" : [
{
"_id" : "oranges",
"numberOf" : 2,
"Total Fruit" : 3
},
{
"_id" : "apples",
"numberOf" : 2,
"Total Fruit" : 1
}
],
"ok" : 1
}