def fetch_holidays_in_date_range(src):
query = "SELECT * from holiday_tab where id = src"
db = dbconnect.connect()
# defining the cursor for reading data
cursor = db.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
# query the database
cursor.execute(query.format(src));
rows = cursor.fetchall()
dbconnect.destroy(cursor, db)
return rows
Could someone help, how to mock this code in pytest or in unittest. I've googled for mock db and I hardly found anything.
Pytest doesn't support you to run the test on production/main DB if you are using pytest-django.
There is a better approach to solve this issue.
Pytest DB resolve method
This says that whenever you run the test with a marker #pytest.mark.django_db, tests are run on another newly created db with name test_your production_db_name.
So if your db name is hello, pytest will create a new db called test_hello and runs tests on it
i am able to run mongodb in my apple console, and vim the mongo.log
right now, i just want to open up the mongodb console so that i can test queries in the console just like the examples in http://www.mongodb.org/display/DOCS/Tutorial
at the moment, the cursor is not returned:
> mongodb
all output going to :/usr/local/var/log/mongodb/mongo.log
and the cursor is not returned. i was expecting the cursor to be returned to so i can do the following :
> mongodb
all output going to :/usr/local/var/log/mongodb/mongo.log
> test = {name : "bouncingHippo"}
> db.family.save(test)
> "ok"
What am i doing wrong? i am using mongoose
I'm not entirely clear which console you are getting this output from, as the Node console won't return anything usable if you just enter mongodb.
If what you are trying to do is just launch a MongoDB console, you will need to first launch the mongod process and then attach to that process with the MongoDB console. The MongoDB console is called mongo. In the simplest test, you can launch mongod from one terminal window and then mongo from another. In the terminal window that is running mongo you can then work through the examples in the tutorial. Your pseudo code would then look like:
MongoDB shell version: 2.2.1
connecting to: 127.0.0.1:18070/test
> test = {name : "bouncingHippo"}
{ "name" : "bouncingHippo" }
> db.family.save(test)
If you are trying to use Mongoose for the pseudo code you have in your question, it would be more like the following from the Node console (assuming Node.js and Mongoose are installed)
var mongoose = require('mongoose');
var db = mongoose.createConnection('mongodb://localhost/test');
var testSchema = new mongoose.Schema({
name: String
})
var Test = db.model('Test', testSchema)
var test = new Test({ name: 'bouncinghippo' })
test.save()
Is there a simple way to do this?
The best way is to do a mongodump then mongorestore. You can select the collection via:
mongodump -d some_database -c some_collection
[Optionally, zip the dump (zip some_database.zip some_database/* -r) and scp it elsewhere]
Then restore it:
mongorestore -d some_other_db -c some_or_other_collection dump/some_collection.bson
Existing data in some_or_other_collection will be preserved. That way you can "append" a collection from one database to another.
Prior to version 2.4.3, you will also need to add back your indexes after you copy over your data. Starting with 2.4.3, this process is automatic, and you can disable it with --noIndexRestore.
At the moment there is no command in MongoDB that would do this. Please note the JIRA ticket with related feature request.
You could do something like:
db.<collection_name>.find().forEach(function(d){ db.getSiblingDB('<new_database>')['<collection_name>'].insert(d); });
Please note that with this, the two databases would need to share the same mongod for this to work.
Besides this, you can do a mongodump of a collection from one database and then mongorestore the collection to the other database.
Actually, there is a command to move a collection from one database to another. It's just not called "move" or "copy".
To copy a collection, you can clone it on the same database, then move the cloned collection.
To clone:
> use db1
switched to db db1
> db.source_collection.find().forEach(
function(x){
db.collection_copy.insert(x)
}
);
To move:
> use admin
switched to db admin
> db.runCommand(
{
renameCollection: 'db1.source_collection',
to : 'db2.target_collection'
}
);
The other answers are better for copying the collection, but this is especially useful if you're looking to move it.
I would abuse the connect function in mongo cli mongo doc. so that means you can start one or more connection.
if you want to copy customer collection from test to test2 in same server. first you start mongo shell
use test
var db2 = connect('localhost:27017/test2')
do a normal find and copy the first 20 record to test2.
db.customer.find().limit(20).forEach(function(p) { db2.customer.insert(p); });
or filter by some criteria
db.customer.find({"active": 1}).forEach(function(p) { db2.customer.insert(p); });
just change the localhost to IP or hostname to connect to remote server. I use this to copy test data to a test database for testing.
If between two remote mongod instances, use
{ cloneCollection: "<collection>", from: "<hostname>", query: { <query> }, copyIndexes: <true|false> }
See http://docs.mongodb.org/manual/reference/command/cloneCollection/
I'd usually do:
use sourcedatabase;
var docs=db.sourcetable.find();
use targetdatabase;
docs.forEach(function(doc) { db.targettable.insert(doc); });
for huge size collections, you can use Bulk.insert()
var bulk = db.getSiblingDB(dbName)[targetCollectionName].initializeUnorderedBulkOp();
db.getCollection(sourceCollectionName).find().forEach(function (d) {
bulk.insert(d);
});
bulk.execute();
This will save a lot of time.
In my case, I'm copying collection with 1219 documents: iter vs Bulk (67 secs vs 3 secs)
Unbelievable how many up-votes are given for agonizingly slow one-by-one copy of data.
As given in other answers the fastest solution should be mongodump / mongorestore. There is no need to save the dump to your local disk, you can pipe the dump directly into mongorestore:
mongodump --db=some_database --collection=some_collection --archive=- | mongorestore --nsFrom="some_database.some_collection" --nsTo="some_or_other_database.some_or_other_collection" --archive=-
In case you run a sharded cluster, the new collection is not sharded by default. All data is written initially to your primary shard. This may cause problems with disk space and put additional load to your cluster for balancing. Better pre-split your collection like this before you import the data:
sh.shardCollection("same_or_other_database.same_or_other_collection", { <shard_key>: 1 });
db.getSiblingDB("config").getCollection("chunks").aggregate([
{ $match: { ns: "some_database.some_collection"} },
{ $sort: { min: 1 } },
{ $skip: 1 }
], { allowDiskUse: true }).forEach(function (chunk) {
sh.splitAt("same_or_other_database.same_or_other_collection", chunk.min)
})
There are different ways to do the collection copy. Note the copy can happen in the same database, different database, sharded database or mongod instances. Some of the tools can be efficient for large sized collection copying.
Aggregation with $merge:
Writes the results of the aggregation pipeline to a specified collection. Note that the copy can happen across databases, even the sharded collections. Creates a new one or replaces an existing collection. New in version 4.2.
Example: db.test.aggregate([ { $merge: { db: "newdb", coll: "newcoll" }} ])
Aggregation with $out:
Writes the results of the aggregation pipeline to a specified collection. Note that the copy can happen within the same database only. Creates a new one or replaces an existing collection.
Example: db.test.aggregate([ { $out: "newcoll" } ])
mongoexport and mongoimport:
These are command-line tools.
mongoexport produces a JSON or CSV export of collection data. The output from the export is used as the source for the destination collection using the mongoimport.
mongodump and mongorestore:
These are command-line tools.
mongodump utility is for creating a binary export of the contents of a database or a collection. The mongorestore program loads data from a binary database dump created by mongodump into the destination.
db.cloneCollection():
Copies a collection from a remote mongod instance to the current mongod instance.
Deprecated since version 4.2.
db.collection.copyTo():
Copies all documents from collection into new a Collection (within the same database).
Deprecated since version 3.0. Starting in version 4.2, MongoDB this command is not valid.
NOTE: Unless said the above commands run from mongo shell.
Reference: The MongoDB Manual.
You can also use a favorite programming language (e.g., Java) or environment (e.g., NodeJS) using appropriate driver software to write a program to perform the copy - this might involve using find and insert operations or another method. This find-insert can be performed from the mongo shell too.
You can also do the collection copy using GUI programs like MongoDB Compass.
You can use aggregation framework to resolve your issue
db.oldCollection.aggregate([{$out : "newCollection"}])
It should be noted, that indexes from oldCollection will not copied in newCollection.
I know this question has been answered however I personally would not do #JasonMcCays answer due to the fact that cursors stream and this could cause an infinite cursor loop if the collection is still being used. Instead I would use a snapshot():
http://www.mongodb.org/display/DOCS/How+to+do+Snapshotted+Queries+in+the+Mongo+Database
#bens answer is also a good one and works well for hot backups of collections not only that but mongorestore does not need to share the same mongod.
This might be just a special case, but for a collection of 100k documents with two random string fields (length is 15-20 chars), using a dumb mapreduce is almost twice as fast as find-insert/copyTo:
db.coll.mapReduce(function() { emit(this._id, this); }, function(k,vs) { return vs[0]; }, { out : "coll2" })
Using pymongo, you need to have both databases on same mongod, I did the following:
db = original database
db2 = database to be copied to
cursor = db["<collection to copy from>"].find()
for data in cursor:
db2["<new collection>"].insert(data)
If RAM is not an issue using insertMany is way faster than forEach loop.
var db1 = connect('<ip_1>:<port_1>/<db_name_1>')
var db2 = connect('<ip_2>:<port_2>/<db_name_2>')
var _list = db1.getCollection('collection_to_copy_from').find({})
db2.collection_to_copy_to.insertMany(_list.toArray())
This won't solve your problem but the mongodb shell has a copyTo method that copies a collection into another one in the same database:
db.mycoll.copyTo('my_other_collection');
It also translates from BSON to JSON, so mongodump/mongorestore are the best way to go, as others have said.
Many right answers here. I would go for mongodump and mongorestore in a piped fashion for a large collection:
mongodump --db fromDB --gzip --archive | mongorestore --drop --gzip --archive --nsFrom "fromDB.collectionName" --nsTo "toDB.collectionName"
although if I want to do quick copy, its slow but it works:
use fromDB
db.collectionName.find().forEach(function(x){
db.getSiblingDB('toDB')['collectionName'].insert(x);
});"
In case some heroku users stumble here and like me want to copy some data from staging database to the production database or vice versa here's how you do it very conveniently (N.B. I hope there's no typos in there, can't check it atm., I'll try confirm the validity of the code asap):
to_app="The name of the app you want to migrate data to"
from_app="The name of the app you want to migrate data from"
collection="the collection you want to copy"
mongohq_url=`heroku config:get --app "$to_app" MONGOHQ_URL`
parts=(`echo $mongohq_url | sed "s_mongodb://heroku:__" | sed "s_[#/]_ _g"`)
to_token=${parts[0]}; to_url=${parts[1]}; to_db=${parts[2]}
mongohq_url=`heroku config:get --app "$from_app" MONGOHQ_URL`
parts=(`echo $mongohq_url | sed "s_mongodb://heroku:__" | sed "s_[#/]_ _g"`)
from_token=${parts[0]}; from_url=${parts[1]}; from_db=${parts[2]}
mongodump -h "$from_url" -u heroku -d "$from_db" -p"$from_token" -c "$collection" -o col_dump
mongorestore -h "$prod_url" -u heroku -d "$to_app" -p"$to_token" --dir col_dump/"$col_dump"/$collection".bson -c "$collection"
You can always use Robomongo. As of v0.8.3 there is a tool that can do this by right-clicking on the collection and selecting "Copy Collection to Database"
For details, see http://blog.robomongo.org/whats-new-in-robomongo-0-8-3/
This feature was removed in 0.8.5 due to its buggy nature so you will have to use 0.8.3 or 0.8.4 if you want to try it out.
use "Studio3T for MongoDB"
that have Export and Import tools by click on database , collections or specific collection
download link : https://studio3t.com/download/
The simplest way to import data from the existing MongoDB atlas cluster DB is using mongodump & mongorestore commands.
To create the dump from existing DB you can use:
mongodump --uri="<connection-uri>"
There are other options for connection which can be lookup here: https://www.mongodb.com/docs/database-tools/mongodump/
After the dump is successfully created in a dump/ directory, you can use import that data inside your other db like so:
mongorestore --uri="<connection-uri-of-other-db>" <dump-file-location>
Similarly for mongorestore, there are other connection options that can be looked up along with commands to restore specific collections:
https://www.mongodb.com/docs/database-tools/mongorestore/
The dump file location will be inside the dump directory. There may be a subdirectory with the same name as DB name which you dumped. For example if you dumped test DB, then dump file location would be /dump/test
In my case, I had to use a subset of attributes from the old collection in my new collection. So I ended up choosing those attributes while calling insert on the new collection.
db.<sourceColl>.find().forEach(function(doc) {
db.<newColl>.insert({
"new_field1":doc.field1,
"new_field2":doc.field2,
....
})
});`
To copy a collection (myCollection1) from one database to another in MongoDB,
**Server1:**
myHost1.com
myDbUser1
myDbPasword1
myDb1
myCollection1
outputfile:
myfile.json
**Server2:**
myHost2.com
myDbUser2
myDbPasword2
myDb2
myCollection2
you can do this:
mongoexport --host myHost1.com --db myDb1 -u myDbUser1 -p myDbPasword1 --collection myCollection1 --out myfile.json
then:
mongoimport --host myHost2.com --db myDb2 -u myDbUser2 -p myDbPasword2 --collection myCollection2 --file myfile.json
Another case , using CSV file:
Server1:
myHost1.com
myDbUser1
myDbPasword1
myDb1
myCollection1
fields.txt
fieldName1
fieldName2
outputfile:
myfile.csv
Server2:
myHost2.com
myDbUser2
myDbPasword2
myDb2
myCollection2
you can do this:
mongoexport --host myHost1.com --db myDb1 -u myDbUser1 -p myDbPasword1 --collection myCollection1 --out myfile.csv --type=csv
add clolumn types in csv file (name1.decimal(),name1.string()..) and then:
mongoimport --host myHost2.com --db myDb2 -u myDbUser2 -p myDbPasword2 --collection myCollection2 --file myfile.csv --type csv --headerline --columnsHaveTypes
This can be done using Mongo's db.copyDatabase method:
db.copyDatabase(fromdb, todb, fromhost, username, password)
Reference: http://docs.mongodb.org/manual/reference/method/db.copyDatabase/