I have two mongodb collections items and settings.
When documents are inserted into items, one of the fields level needs to be read from the settings table. Currently I'm running two queries to make this happen.
Is there a way to do this as one query like I could do with SQL with select queries inside an insert?
// read
let settingRecord = connection.collection("settings").findOne({
"group_id": 2
}, function(err, result) {});
// write
connection.collection("items").insertOne({
"item_id": 2,
"group_id": 2,
"level": settingRecord['level']
}, function(err, result) {});
You can do as below using $merge but with some more pipelines as mentioned by #D. SM
db.getCollection('test').aggregate( [
{ $match : { group: 2 } },
{ $project: {level: 1, group:1} },
{ $addFields:{item:2} },
{ $merge : { into: { db: "local", coll: "test2" }, whenMatched: [{ $addFields: {
"level":"$$new.level", "group":"$$new.group", "item":"$$new.item"
} } ] , whenNotMatched: "insert" }}
] )
Explanation:
{ $match : { group: 2 } }
Match on a field to get the document - In your case, it would be settings collection
{ $project: {level: 1, group:1} },
Project only the required fields to the next stage. You can skip this stage.
{ $addFields:{item:2} },
Using this you can add list of new fields to be inserted.
Actual magic happens in $merge
into - db and collection where data has to be inserted
whenMatched - add what fields to be inserted, $$new is used to refer the fields matched in the previous stage
Refer this
You can play with the pipelines to avoid unnecessary things. $merge has more options. In this example, you can skip whenNotMatched if you don't want to insert for non match items. You can specify to fail in that case.
I could merge into a new collection with a merge command for the given requirement of updating the level field. Let me know how to update the same collection as in aggregate collection eg. items collection in this case.
> db.newCollection.find();
{ "_id" : 1, "groupid" : 2, "itemid" : 2, "settingsDoc" : [ { "level" : 10 } ] }
{ "_id" : 2, "groupid" : 3, "itemid" : 3, "settingsDoc" : [ { "level" : 30 } ] }
> db.items1.find();
{ "_id" : 1, "groupid" : 2, "itemid" : 2, "level" : null }
{ "_id" : 2, "groupid" : 3, "itemid" : 3, "level" : null }
> db.settings.find();
{ "_id" : 1, "groupid" : 2, "level" : 10 }
{ "_id" : 2, "groupid" : 3, "level" : 30 }
> db.items1.aggregate([
... {$lookup:{
... from: "settings",
... localField: "groupid",
... foreignField: "groupid",
... as:"settingsDoc"
... }
... },
... {$project:{
... groupid:1,
... itemid:1,
... "settingsDoc.level":1
... }
... },
... {$merge:{into:"newCollection"}
... }
... ]);
> db.newCollection.find();
{ "_id" : 1, "groupid" : 2, "itemid" : 2, "settingsDoc" : [ { "level" : 10 } ] }
{ "_id" : 2, "groupid" : 3, "itemid" : 3, "settingsDoc" : [ { "level" : 30 } ] }
>
I am running an IN query against my collection.
Here is my query structure:
db.myCollection.find (
{ deviceId : { $in : [ "ABC", "XYZ" .. ] }
})
I know for a fact that each of these will return me multiple rows, but I am only interested in getting the first result of each.
I looked into the $first aggregation function, but could not see it a fit for my situation.
I could do a findOne query against each ID one at a time and combine the results in my client code written in Java. But there can be a lot of these IDs and I want to reduce the network round trip each time.
EDIT. Adding an example for more clarity.
Sample data in my collection:
{ "_id" : 1, "deviceSerial" : 1, "deviceId" : "ABC" }
{ "_id" : 2, "deviceSerial" : 2, "deviceId" : "XYZ" }
{ "_id" : 3, "deviceSerial" : 3, "deviceId" : "LMN" }
{ "_id" : 4, "deviceSerial" : 4, "deviceId" : "PQR" }
{ "_id" : 5, "deviceSerial" : 5, "deviceId" : "SDS" }
{ "_id" : 6, "deviceSerial" : 6, "deviceId" : "KLP" }
Now, if I do my query with { deviceId : { $in : [ "LMN", "XYZ" ] }
Expected Output (sort does not matter):
{ "_id" : 2, "deviceSerial" : 2, "deviceId" : "XYZ" }
{ "_id" : 3, "deviceSerial" : 3, "deviceId" : "LMN" }
So the idea with $first is good. You need to filter the collection with $in and then eliminate duplicates. Following aggregation should work:
db.myCollection.aggregate([
{
$match: {
deviceId: { $in: ["ABC", "XYZ"] }
}
},
{
$group: {
_id: "$deviceId",
doc: { "$first": "$$CURRENT" }
}
},
{
$replaceRoot: { newRoot: "$doc" }
}
])
doc will store first entire document for each group. On the last stage we need to promote this document to be a root and $replaceRoot is able to do that.
I have a collection called constructora that has the following structure:
{
"_id" : A,
"edificio" : [
{
"_id": X,
"a" : 48,
"b" : 59
},
{
"_id": Y,
"a" : 8,
"b" : 5
},
{
"_id": Z,
"a" : 0,
"b" : -1
},
...
]
}
So, I want to make a query that returns, for each sub document (edificio) his parent's _id. An example:
{
"_id" : X,
"a" : 48,
"b" : 59
"id_constructora" : A
}
{
"_id" : Y,
"a" : 8,
"b" : 5
"id_constructora" : A
}
{
"_id" : Z,
"a" : 0,
"b" : -1
"id_constructora" : A
}
How can I do that?
EDIT
Now I'm trying using aggregate, and grouping my query by "edificio_id", so for each document in edificio I can get my desired output:
db.constructora.aggregate(
[
{ $project : { "_id" : 1, "edificio._id" : 1 } },
{ $group : { _id : "$edificio._id" } }
]
).pretty();
But it doesn't work. The output is:
...
{
"_id" : [
ObjectId("613339376430333562373466"),
ObjectId("663736363935393066656236"),
ObjectId("313933613036363364633832"),
ObjectId("653135313831633638336436")
]
}
{
"_id" : [
ObjectId("643531326231663739626465"),
ObjectId("343231386237333365356461"),
ObjectId("373461303864636138393263"),
ObjectId("386433623966653737343962"),
ObjectId("303863633366376431363335"),
ObjectId("663833343161643639376161"),
ObjectId("383833363836663532633733"),
ObjectId("396330313961353137333166"),
ObjectId("646535366662363364613837"),
ObjectId("633937613032656436653965")
]
}
You can use $unwind to break the embedded array into embedded docs, $addFields to rename and add the _id into the embedded doc followed by $replaceRoot to promote the embedded document to the top level in 3.4 mongo server.
db.constructora.aggregate([
{$unwind:"$edificio"},
{$addFields:{"edificio.id_constructora":"$_id"}},
{$replaceRoot: {newRoot: "$edificio"}}
])
More info here https://docs.mongodb.com/manual/reference/operator/aggregation/replaceRoot/#replaceroot-with-an-array-element
I have a MovieRatings database with columns userId, movieId, movie-categoryId, reviewId, movieRating and reviewDate.
In my mapper I want to extract userId -> (movieId, movieRating)
And then in the reducer I want to group all movieId, movieRating pair by user.
Here is my attempt:
Map function:
var map = function() {
var values={movieId : this.movieId, movieRating : this.movieRating};
emit(this.userId, values);}
Reduce function:
var reduce = function(key,values) {
var ratings = [];
values.forEach(function(V){
var temp = {movieId : V.movieId, movieRating : V.movieRating};
Array.prototype.push.apply(ratings, temp);
});
return {userId : key, ratings : ratings };
}
Run MapReduce:
db.ratings.mapReduce(map, reduce, { out: "map_reduce_step1" })
Output: db.map_reduce_step1.find()
{ "_id" : 1, "value" : { "userId" : 1, "ratings" : [ ] } }
{ "_id" : 2, "value" : { "userId" : 2, "ratings" : [ ] } }
{ "_id" : 3, "value" : { "userId" : 3, "ratings" : [ ] } }
{ "_id" : 4, "value" : { "userId" : 4, "ratings" : [ ] } }
{ "_id" : 5, "value" : { "userId" : 5, "ratings" : [ ] } }
{ "_id" : 6, "value" : { "userId" : 6, "ratings" : [ ] } }
{ "_id" : 7, "value" : { "userId" : 7, "ratings" : [ ] } }
{ "_id" : 8, "value" : { "userId" : 8, "ratings" : [ ] } }
{ "_id" : 9, "value" : { "userId" : 9, "ratings" : [ ] } }
{ "_id" : 10, "value" : { "userId" : 10, "ratings" : [ ] } }
{ "_id" : 11, "value" : { "userId" : 11, "ratings" : [ ] } }
{ "_id" : 12, "value" : { "userId" : 12, "ratings" : [ ] } }
{ "_id" : 13, "value" : { "userId" : 13, "ratings" : [ ] } }
{ "_id" : 14, "value" : { "userId" : 14, "ratings" : [ ] } }
{ "_id" : 15, "value" : { "movieId" : 1, "movieRating" : 3 } }
{ "_id" : 16, "value" : { "userId" : 16, "ratings" : [ ] } }
I am not getting the expected output. In fact, this output makes no sense to me!
Here is the python equivalent of what I am trying to do in the reducer (just in case the purpose of reducer wasn't clear above) :
def reducer_ratings_by_user(self, user_id, itemRatings):
#Group (item, rating) pairs by userID
ratings = []
for movieID, rating in itemRatings:
ratings.append((movieID, rating))
yield user_id, ratings
Edit 1 #chridam
Here is an outline of what I really want to do here :
Movies.csv file looks like :
userId,movieId,movie-categoryId,reviewId,movieRating,reviewDate
1,1,1,1,5,7/12/2000
2,1,1,2,5,7/12/2000
3,1,1,3,5,7/12/2000
4,1,1,4,4,7/12/2000
5,1,1,5,4,7/12/2000
6,1,1,6,5,7/15/2000
1,2,1,7,4,7/25/2000
8,1,1,8,4,7/28/2000
9,1,1,9,3,8/3/2000
...
...
I import this into mongoDB :
mongoimport --db SomeName --collection ratings --type csv --headerline --file Movies.csv
Then I am trying to apply the map-reduce function as define above. After that I will export it back to a csv by doing somethig like :
mongoexport --db SomeName --collection map_reduce_step1 --csv --out movie_ratings_out.csv --fields ...
This movie_ratings_out.csv file should be like :
userId, movieId1, rating1, movieId2, rating2 ,...
1,1,5,2,4
...
...
So each row contains all the (movie,rating) pair for every user.
Edit 2
Sample :
db.ratings.find().pretty()
{
"_id" : ObjectId("57f4a0dd9cb74fc4d344a40f"),
"userId" : 4,
"movieId" : 1,
"movie-categoryId" : 1,
"reviewId" : 4,
"movieRating" : 4,
"reviewDate" : "7/12/2000"
}
{
"_id" : ObjectId("57f4a0dd9cb74fc4d344a410"),
"userId" : 5,
"movieId" : 1,
"movie-categoryId" : 1,
"reviewId" : 5,
"movieRating" : 4,
"reviewDate" : "7/12/2000"
}
{
"_id" : ObjectId("57f4a0dd9cb74fc4d344a411"),
"userId" : 4,
"movieId" : 2,
"movie-categoryId" : 1,
"reviewId" : 6,
"movieRating" : 5,
"reviewDate" : "7/15/2000"
}
{
"_id" : ObjectId("57f4a0dd9cb74fc4d344a412"),
"userId" : 4,
"movieId" : 3,
"movie-categoryId" : 1,
"reviewId" : 2,
"movieRating" : 5,
"reviewDate" : "7/12/2000"
}
...
Then after MapReduce expected output json is :
{
"_id" : ....,
"userId" : 4,
"movieList" : [ {
"movieId" : 2
"movieRating" : 5
},
{
"movieId" : 1
"movieRating" : 4
}
...
]
}
{
"_id" : ....,
"userId" : 5,
"movieList" : ...
}
...
You just need to run an aggregation pipeline which consists of a $group stage that summarize documents. This groups input documents by a specified identifier expression and applies the accumulator expression(s). The $group pipeline operator is similar to the SQL's GROUP BY clause. In SQL, you can't use GROUP BY unless you use any of the aggregation functions. The same way, you have to use an aggregation function in MongoDB as well. You can read more about the aggregation functions here.
The accumulator operator you would need to create the movieList array is $push.
Another pipeline which follows after the $group stage is the $project operator which is used to select or reshape each document in the stream, include, exclude or rename fields, inject computed fields, create sub-document fields, using mathematical expressions, dates, strings and/or logical (comparison, boolean, control) expressions - similar to what you would do with the SQL SELECT clause.
The last step is the $out pipeline which writes the resulting documents of the aggregation pipeline to a collection. It must be the last stage in the pipeline.
So as a result, you can run the following aggregate operation:
db.ratings.aggregate([
{
"$group": {
"_id": "$userId",
"movieList": {
"$push": {
"movieId": "$movieId",
"movieRating": "$movieRating",
}
}
}
},
{
"$project": {
"_id": 0, "userId": "$_id", "movieList": 1
}
},
{ "$out": "movie_ratings_out" }
])
Using the sample 5 documents above, the sample output if you query db.getCollection('movie_ratings_out').find({}) would yield:
/* 1 */
{
"_id" : ObjectId("57f52636b9c3ea346ab1d399"),
"movieList" : [
{
"movieId" : 1.0,
"movieRating" : 4.0
}
],
"userId" : 5.0
}
/* 2 */
{
"_id" : ObjectId("57f52636b9c3ea346ab1d39a"),
"movieList" : [
{
"movieId" : 1.0,
"movieRating" : 4.0
},
{
"movieId" : 2.0,
"movieRating" : 5.0
},
{
"movieId" : 3.0,
"movieRating" : 5.0
}
],
"userId" : 4.0
}
I have a simple structured document like this:
"people" : [
{
"id" : "6241863",
"amount" : 5
}
],
People can contain more than one element. I've managed to get this to work:
db.village.findOne({"people": {"$in": [{"id": "6241863", "amount": 5}]}})
But I want to ignore amount and search for any document containing people with id 6241863 and any amount.
According to the advanced query documentation, you can mix array value queries with dot notation for reaching into objects. Here's an example using your schema:
$ mongo
MongoDB shell version: 2.1.0
connecting to: test
> db.users.save({_id: 1, friends: [{id: 2, name: 'bob'}]})
> db.users.find({friends: {id: 2, name: 'bob'}})
{ "_id" : 1, "friends" : [ { "id" : 2, "name" : "bob" } ] }
> db.users.find({'friends.id': 2})
{ "_id" : 1, "friends" : [ { "id" : 2, "name" : "bob" } ] }
> db.users.find({'friends.name': 'bob'})
{ "_id" : 1, "friends" : [ { "id" : 2, "name" : "bob" } ] }
> db.users.find({'friends.name': 'ted'})
>
Try with that
db.village.findOne({"people" : { "$in" : [{ "id" : "6241863" , "amount" : { "$ne" : null } }]}})