Saving the result of a MongoDB query - mongodb

When doing a research in mongo shell I often write quite complex queries and want the result to be stored in other collection. I know the way to do it with .forEach():
db.documents.find(query).forEach(function(d){db.results.insert(d)})
But it's kind of tedious to write that stuff each time. Is there a cleaner way? I'd like the syntax to be something like db.documents.find(query).dumpTo('collectionName').

Here's a solution I'll use: db.results.insert(db.docs.find(...).toArray())
There is still too much noise, though.
UPD: There is also an option to rewrite find using aggregation pipeline. Then you can use $out operator.

It looks like you are doing your queries from the mongo shell, which allows you to write code in javascript. You can assign the result of queries to a variable:
result = db.mycollection.findOne(my_query)
And save the result to another collection:
db.result.save(result)
You might have to remove the _id of the result if you want to append it to the result collection, to prevent a duplicate key error
Edit:
db.mycollection.findOne({'_id':db.mycollection.findOne()['_id']})
db.foo.save(db.bar.findOne(...))
If you want to save an array, you can write a javascript function. Something like the following should work (I haven't tested it):
function save_array(arr) {
for(var i = 0; i < arr.length; i++) {
db.result.save(arr[i])
}
}
...
result = db.mycollection.find(...)
save_array(result)
If you want the function to be available every time you start mongo shell, you can include it in your .mongorc.js file

As far as I know, there isn't built-in functionality to do this in MongoDB.
Other options would be to use mongoexport/mongoimport or mongodump/mongorestore functionalities.
In both mongoexport and mongodump you can filter the results by adding query options using --query <JSON> or -q <JSON>.

If your query is using an aggregation operator then the solution is as sample as using the $out.
I created a sample Collection with the name "tester" which contain the following records.
{ "_id" : ObjectId("4fb36bfd3d1c88bfa15103b1"), "name" : "bob", "value" : 5, "state" : "b"}
{ "_id" : ObjectId("4fb36c033d1c88bfa15103b2"), "name" : "bob", "value" : 3, "state" : "a"}
{ "_id" : ObjectId("4fb36c063d1c88bfa15103b3"), "name" : "bob", "value" : 7, "state" : "a"}
{ "_id" : ObjectId("4fb36c0c3d1c88bfa1a03b4"), "name" : "john", "value" : 2, "state" : "a"}
{ "_id" : ObjectId("4fb36c103d1c88bfa5103b5"), "name" : "john", "value" : 4, "state" : "b"}
{ "_id" : ObjectId("4fb36c143d1c88bfa15103b"), "name" : "john", "value" : 8, "state" : "b"}
{ "_id" : ObjectId("4fb36c163d1c88bfa15103a"), "name" : "john", "value" : 6, "state" : "a"}
Now using the aggregate operator I perform a group by and then save the result into a new collection using this magical operator "$out".
db.tester.aggregate([{$group:{
_id:{name:"$name",state:"$state"},
min:{$min:"$value"},
max:{$max:"$value"},
} },
{$out:"tester_max_min"}
])
What basically the query is trying to do is, group by name & state and find the min and max values for each individual group, and then save the result into a new collection named "tester_max_min"
db.tester_max_min.find();
The new collection formed will have the following documents in it :
{ "_id" : { "name" : "john", "state" : "b" }, "min" : 4, "max" : 8 }
{ "_id" : { "name" : "john", "state" : "a" }, "min" : 2, "max" : 6 }
{ "_id" : { "name" : "bob", "state" : "a" }, "min" : 3, "max" : 7 }
{ "_id" : { "name" : "bob", "state" : "b" }, "min" : 5, "max" : 5 }
I still need to explore how helpful can $out is but it works like a charm for any aggregator operator.

Related

How create a MongoDB query

I have a collection with some documents like below:
{
"_id" : ObjectId("1"),
"className" : "model.MyClass",
"createdOn" : ISODate("2018-10-23T11:00:00.000+01:00"),
"status" : "A"
}
{
"_id" : ObjectId("2"),
"className" : "model.MyClass",
"createdOn" : ISODate("2018-10-23T11:01:00.000+01:00"),
"status" : "B"
}
{
"_id" : ObjectId("3"),
"className" : "model.MyClass",
"createdOn" : ISODate("2018-10-23T11:02:00.000+01:00"),
"status" : "C"
}
{
"_id" : ObjectId("4"),
"className" : "model.MyClass",
"createdOn" : ISODate("2018-10-23T11:03:00.000+01:00"),
"status" : "D"
}
Given a specific ID, how can I get the previous document that whose status not equals a specific status.
For example, I give the ID 4 and like to get the last document that status not is B neither C. So, I get the Object with Id 1.
How to create this query?
you could try this:
db.yourcollection.find( {"status":{"$nin":["B","C"]}}
).sort({_id:-1}).limit(1);
so use not in operator i.e. $nin, then sort the data in descending order and limit the records to 1
see below documentations for details.
$nin operator
mongo sort

mongodb count and remove duplicate values

i have a large mongodb collection with a lot of duplicate inserts like this
{ "_id" : 1, "val" : "222222", "val2" : "37"}
{ "_id" : 2, "val" : "222222", "val2" : "37" }
{ "_id" : 3, "val" : "222222", "val2" : "37" }
{ "_id" : 4, "val" : "333333", "val2" : "66" }
{ "_id" : 5, "val" : "111111", "val2" : "22" }
{ "_id" : 6, "val" : "111111", "val2" : "22" }
{ "_id" : 7, "val" : "111111", "val2" : "22" }
{ "_id" : 8, "val" : "111111", "val2" : "22" }
i want to count all duplicates for each insert and only leave one unique entry with the count number in DB like this
{ "_id" : 1, "val" : "222222", "val2" : "37", "count" : "3"}
{ "_id" : 2, "val" : "333333", "val2" : "66", "count" : "1"}
{ "_id" : 2, "val" : "111111", "val2" : "22", "count" : "4" }
i already checked out MapReduce and aggregation framework but they never output the full document back and only do one calculation for full collection
it would be good to save the new data to a new collection
If you use mongodb 2.6, here is an example with the aggregation framework :
db.duplicate.aggregate({$group:{_id:"$val",count:{$sum :1}}},
{$project:{_id:0, val:"$_id", count:1}},
{$out:"deduplicate"})
group with val and count
project to rename _id field and mask _id field
out to write to a new collection (here the name is deduplicate)
Hope it fit with your case.
Might be easier with an incremental map reduce
mapper=function(){
emit({'val1':this.val, 'val2':this.val2}, {'count':1});
}
reducer=function(k,v){
counter=0;
for (i=0;i<v.length;i++){
counter+=v[i].count;
}
return {'count':counter}
}
Then in the shell you'll need to do
bigcollection.map_reduce(mapper, reducer, {out:{reduce:'reducedcollection'}})
This should result in a new collection called reduced collection. Your values will be the IDs and the count will be there. Note the use of two values as the key in your new collection. If you want to find a specific instance you can do:
reducedcollection.findOne({'id.val1':'33333', 'id.val2':'22'})
The interesting thing happens is that you can now drop the old collection and as new data comes in, map reduce it on top of the reducedcollection and you'll increment the counts.
Might be handy?

Morphia - Query based on a subdocument

I have a mongo collection which has documents which look like below :
{
"_id" : ObjectId("9873214jkhdkfjdsf8324"),
"nm" : "test",
"sts" : 1,
"updby" : NumberLong(0),
"tags" : [
{
"name" : "women",
"rank" : 1,
"type" : 3
},
{
"name" : "men",
"rank" : 1
},
{
"name" : "clothing",
"rank" : 2,
"type" : 1
}
]
}
I want to query the collection such that I want all the documents which have "name": "women" and "type" : 3 inside the tags subdocument of each document which is returned.
I know that the mongo query should be something like this :
db.collection.find("tags":{
$all:[
{"$elemMatch":{"name":"women","type":3}},
]})
I tried using the 'hasthiselement' provided by morphia, but I am not able to form the exact query which I want.
getMongoDAORead().getDatastore().createQuery(test.class)
.field("tags").hasThisElement("name").equal("women");
This query doesn't seem to be correct. Can someone help me form the correct query?
I fixed this by doing the following:
I created a object of the Tags Class and initialized it:
Tags tag = new Tags("women", null, 3);
Query<MyClass> t = getMongoDAORead().getDatastore()
.createQuery(MyClass.class)
.field("ctags").hasThisElement(tag);

mongodb part of the array

I have a schema which looks like this:
{
"_id" : ObjectId("4f6af5c7065f92581a000013"),
....
"conversation" : [{
"_id" : ObjectId("4f6af5c7065f92581a000013"),
"msg" : "message1",
"userID" : 1
},{
"_id" : ObjectId("4f6af5c7065f92581a000016"),
"msg" : "message3",
"userID" : 1
},{
"_id" : ObjectId("4f6af5c7065f92581a000023"),
"msg" : "msg",
"userID" : 1
}]
}
What I need is to output a list of elements whose value for the key msg contains 'msg'
Is it possible to do so?
db.dialogs.find({ "_id" : new ObjectId('4f6af5c7065f92581a000013'), "conversation.msg" : /msg/i })
but it outputs all the information about the object. What I need is just
{
"_id" : ObjectId("4f6af5c7065f92581a000023"),
"msg" : "msg",
"userID" : 1
}
No, you can't select elements of sub-array on their own. Structure of parent document will be preserved (though, you may choose not to select from fields other tnan conversation (as you did in the example)).

How to add new filed by type data to existing documents Mongodb?

I am trying to add field(key) existing document all records. And i wanna; when new record saved, automaticly save to current date this field;
Now my one document instance of studen
{
"_id" : ObjectId("4f83ed9bcba9560ea474465d"),
"name" : "",
"lastname" : "",
"number" : "",
.....
}
I want to this;
{
"_id" : ObjectId("4f83ed9bcba9560ea474465d"),
"name" : "",
"lastname" : ""
"Date" : currentTime()
}
When new record saved to document automaticly save current time to documents.
I dont wanna do this from client side,
Just to make sure I am understanding correctly, are you looking to update every document in your collection with a new "Date" field, or are you just looking to add the current date to new documents as they are inserted?
Given a collection (truncated for brevity) like the following:
> db.students.find()
{ "_id" : 1, "name" : "Joe" }
{ "_id" : 2, "name" : "Jane" }
{ "_id" : 3, "name" : "Sam" }
>
A Date may be added to every document in the collection in a single update statement like so:
> db.students.update({}, {$set:{"Date": new Date()}}, false, true)
> db.students.find()
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 1, "name" : "Joe" }
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 2, "name" : "Jane" }
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 3, "name" : "Sam" }
>
The false and true values passed to the update function indicate upsert=false and multi=true. More information on this may be found in the Mongo documentation on the update() function:
http://www.mongodb.org/display/DOCS/Updating#Updating-update%28%29
To add a new Date object to new documents as they are added to the collection, simply create a new Date object when the documents are saved:
> db.students.save({ "_id" : 4, "name" : "Chris", "Date":new Date()})
> db.students.find()
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 1, "name" : "Joe" }
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 2, "name" : "Jane" }
{ "Date" : ISODate("2012-04-10T15:06:34.608Z"), "_id" : 3, "name" : "Sam" }
{ "_id" : 4, "name" : "Chris", "Date" : ISODate("2012-04-10T15:12:03.161Z") }
>
When a Date object is created in Java Script, with no values passed into the constructor, it will create a date object with the current UTC (universal, or GMT) time. More information on the Java Script Date object may be found here:
http://www.w3schools.com/jsref/jsref_obj_date.asp
Different languages handle dates differently, so depending on which language your application is written in, you will have to check that language's documentation on how it implements a date object.
Hopefully this will get you pointed in the right direction.