I have really been puzzled by this one. Below is my database and collections. I can not get mongoexport to dump the collection itunes_itunes_level4_US_uniq into json.
I'm trying this via:
mongoexport -d test-database -c itunes_itunes_level4_US_uniq -o itunes_itunes_level4_US_uniq.json
2016-07-21T19:09:37.507-0500 connected to: localhost
2016-07-21T19:09:37.508-0500 exported 0 records
Same command allows me to export the other collections successfully.
What am I doing wrong?
> show dbs
admin (empty)
local 0.078GB
test 0.078GB
test-database 47.931GB
> show collections
[object Object]
extract
extract_4l
extract_level4
itunes_itunes_level4_US_uniq
itunes_level4_US
rabbit_US_uniq
system.indexes
> use test-database
switched to db test-database
> db.itunes_itunes_level4_US_uniq.stats(1024)
{
"ns" : "test-database.itunes_itunes_level4_US_uniq",
"count" : 986099,
"size" : 5295580,
"avgObjSize" : 5499,
"storageSize" : 6002404,
"numExtents" : 24,
"nindexes" : 1,
"lastExtentSize" : 1818052,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 7,
"indexSizes" : {
"_id_" : 7
},
"ok" : 1
}
mongoexport --db test-database --collection itunes_itunes_level4_US_uniq --out itunes_itunes_level4_US_uniq.json
Export whole db and find your collection in it by following command :
mongodump -d <database name> -o <directory_backup>
use above command.
Related
I'm writing dummy server to insert to mongodb, the connection string already match mongo command line connection string, the database and collection name too. Inserting using programming language (PL) driver works fine, Inserting using mongo shell also works fine. But both record seems doesn't shown each other (records that inserted using PL only can be seen when queried using PL, records that inserted using mongo shell only can be found when queried using mongo command line). What's the possible cause for this?
On programming language (there's bunch of records already, only showing first 2 records)
On mongo shell (only 1 record which I inserted just now)
rs0:PRIMARY> db.stats()
{
"db" : "test",
"collections" : 1,
"views" : 0,
"objects" : 1,
"avgObjSize" : 36,
"dataSize" : 36,
"storageSize" : 24576,
"indexes" : 1,
"indexSize" : 24576,
"totalSize" : 49152,
"scaleFactor" : 1,
"fsUsedSize" : 138781839360,
"fsTotalSize" : 233197473792,
"ok" : 1,
"$clusterTime" : {
"clusterTime" : Timestamp(1598360409, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
},
"operationTime" : Timestamp(1598360409, 1)
}
rs0:PRIMARY> db.test1.points.find()
{ "_id" : ObjectId("5f4506c18fda02c740f2650d"), "test" : 1 }
What's the possible cause of this? was it because I modify the systemd command line argument from
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
to
ExecStart=/usr/bin/mongod --config /etc/mongod.conf --replSet rs0
But this on the server, this should globally effecting both client right?
Nevermind, just know that this is NOT the correct way to access the database
db.dbName.colName
db.test1.points.find()
this is the correct way, I don't know since when this changed or it's already like that since beginning..
db = db.getSiblingDB('dbName')
db = db.getSiblingDB('test1')
db.points.find()
I have deployed the Mongo DB replica set using this template mongodb-replica-set-centos.
Mongo DB VM 1 (primary):
ps aux | grep mongo
root 10161 0.7 0.5 797140 40900 ? SLl 05:18 0:05 mongod --dbpath /var/lib/mongo/ --replSet repset --logpath /var/log/mongodb/mongod.log --fork --config /etc/mongod.conf
sshuser 10347 0.0 0.0 112640 960 pts/0 S+ 05:29 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-03-23T05:18:21.137+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-03-23T05:18:21.137+0000 I CONTROL [initandlisten]
repset:PRIMARY> rs.status()
{
"set" : "repset",
"date" : ISODate("2018-03-23T07:38:45.694Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "52.170.83.3:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 8426,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"electionTime" : Timestamp(1521782318, 1),
"electionDate" : ISODate("2018-03-23T05:18:38Z"),
"configVersion" : 2,
"self" : true
},
{
"_id" : 1,
"name" : "10.0.1.5:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8407,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"lastHeartbeat" : ISODate("2018-03-23T07:38:45.538Z"),
"lastHeartbeatRecv" : ISODate("2018-03-23T07:38:42.546Z"),
"pingMs" : NumberLong(1),
"configVersion" : 2
}
],
"ok" : 1
}
Mongo DB VM 2 (secondary):
ps aux | grep mongo
root 10115 0.4 0.5 447908 37892 ? SLl 05:11 0:17 mongod --dbpath /var/lib/mongo/ --config /etc/mongod.conf --replSet repset --logpath /var/log/mongodb/mongod.log --fork
sshuser 10269 0.0 0.0 112640 960 pts/0 S+ 06:21 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
2018-03-23T07:38:54.311+0000 E QUERY [thread1] Error: Authentication failed. :
DB.prototype._authOrThrow#src/mongo/shell/db.js:1441:20
#(auth):6:1
#(auth):1:2
exception: login failed
Mongo DB VM 3(secondary):-
ps aux | grep mongo
root 10122 0.6 0.5 795472 40420 ? SLl 05:12 0:26 mongod --dbpath /var/lib/mongo/ --config /etc/mongod.conf --replSet repset --logpath /var/log/mongodb/mongod.log --fork
sshuser 10381 0.0 0.0 112640 960 pts/0 S+ 06:21 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-03-23T05:12:19.613+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-03-23T05:12:19.613+0000 I CONTROL [initandlisten]
repset:SECONDARY> rs.status()
{
"set" : "repset",
"date" : ISODate("2018-03-23T07:39:04.009Z"),
"myState" : 2,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "52.170.83.3:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 8425,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"lastHeartbeat" : ISODate("2018-03-23T07:39:02.571Z"),
"lastHeartbeatRecv" : ISODate("2018-03-23T07:39:03.573Z"),
"pingMs" : NumberLong(1),
"electionTime" : Timestamp(1521782318, 1),
"electionDate" : ISODate("2018-03-23T05:18:38Z"),
"configVersion" : 2
},
{
"_id" : 1,
"name" : "10.0.1.5:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8806,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"infoMessage" : "could not find member to sync from",
"configVersion" : 2,
"self" : true
}
],
"ok" : 1
}
Questions:-
Why I am unable to login into Mongo DB VM 2?
After I shutdown Mongo DB VM 3, will Mongo DB VM 2 acts as a secondary node?
If I shutdown Mongo DB VM 1, Will any one of secondary node act as a primary node?
All three questions are answered by the same fact: DB VM2 is not part of the replica set. It's clear from the rs.status() information that only two nodes are registered as part of the replica set, VM1 and VM3.
The implications are that:
On DB VM2, it is not part of the replica set so it does not have the authentication credentials you are trying to log in with
No, DB VM2 will not act as a secondary node; because it is not part of the replica set
In the current setup, with only 2 nodes in the replica set, if you shut down either node (VM1 or VM3) then the other node will not elect itself primary, because it cannot command a majority in an election.
Take a look at the docs on Replica Set Elections to understand what the majority is and why it matters; and take a look at DB VM2 to understand why it is not part of your replica set. Did you ever actually add it?
I am just getting started with mongodb and I'm currently just messing around with the console. I have run both.
> db.something.insert({"cmon": "plz"})
> db.teams.save({country:"England",GroupName:"D"})
They both get a response of WriteResult({ "nInserted" : 1 }).
I can also access the inserted data when running .find(), however I keep getting this when I run >show dbs
admin 0.000GB
local 0.000GB
mydb 0.000GB
Does this mean none of my data is actually persisting? Why would this happen?
Yes it is normal, because you probably don't have much data in the collection. Try this to grab more information on the specific collection.
use mydb
db.stats()
You should get something like this (Here is some arbitrary data):
{
"db" : "mydb",
"collections" : 1,
"objects" : 10,
"avgObjSize" : 237.6,
"dataSize" : 2376,
"storageSize" : 36864,
"numExtents" : 0,
"indexes" : 1,
"indexSize" : 36864,
"ok" : 1
}
When using mongorestore with option --oplogReplay to replay oplogs, I found a strange error that mongorestore cannot handle binData field's set operation. You maybe meet the same error if you do this:
insert a test data.
db.testData.insert({_id: 10000, data: BinData(0, ""), size: 10})
update its binData field.
db.testData.update({_id: 10000}, {$set: {data: BinData(0, "CgxVfs93PiT/DrxMSvASFgoNMTAuMTYwLjIyMi4xMhDEJxgKIAA=")}})
update its other field
db.testData.update({_id: 10000}, {$set: {size: 20}})
check with oplog
use local
db.oplog.rs.find().sort({$natural: -1})
you may see the following response:
{ "ts" : Timestamp(1435627154, 1), "h" : NumberLong("-4979206321598144076"), "v" : 2, "op" : "u", "ns" : "test.testData", "o2" : { "_id" : 10000 }, "o" : { "$set" : { "size" : 20 } } }
{ "ts" : Timestamp(1435627144, 1), "h" : NumberLong("2899524097634687825"), "v" : 2, "op" : "u", "ns" : "test.testData", "o2" : { "_id" : 10000 }, "o" : { "$set" : { "data" : BinData(0,"CgxVfs93PiT/DrxMSvASFgoNMTAuMTYwLjIyMi4xMhDEJxgKIAA=") } } }
{ "ts" : Timestamp(1435627136, 1), "h" : NumberLong("-8486373688715225152"), "v" : 2, "op" : "i", "ns" : "test.testData", "o" : { "_id" : 10000, "data" : BinData(0,""), "size" : 10 } }
dump these two oplog and replay it
In bash shell:
mongodump --port 27017 -d local -c oplog.rs --query '{"ts" : {$gte: Timestamp(1435627144, 1)}}' -o ./oplogD/
mv ./oplogD/local/oplog.rs.bson ./oplogR/oplog.bson
mongorestore --port 27017 --oplogReplay ./oplogR/
after this you would find data not as expected. In my own, data changes to this.
{ "_id" : 10000, "data" : BinData(0,"ADRAAAAAPiT/DrxMSvASFgoNMTAuMTYwLjIyMi4xMhDEJxgKIAA="), "size" : 20 }
The size field is really correct, but the data field is not correct.
The most strange thing would be this, if you dump only one oplog and replay it, the data would be correct.
mongodump --port 27017 -d local -c oplog.rs --query '{"ts" : Timestamp(1435627144, 1)}' -o ./tmpD/
mv ./tmpD/local/oplog.rs.bson ./tmpR/oplog.bson
mongorestore --port 27017 --oplogReplay ./tmpR/
After oplog replayed, the 'data' field is quite correct.
{ "_id" : 10000, "data" : BinData(0,"CgxVfs93PiT/DrxMSvASFgoNMTAuMTYwLjIyMi4xMhDEJxgKIAA="), "size" : 10 }
Why does this strange thing happen?
It was fixed in this commit.
https://github.com/mongodb/mongo-tools/commit/ed60bbfae7d2b5239bea69f162f0784e17995e91
Trace the bug report in JIRA.
https://jira.mongodb.org/browse/TOOLS-807
Is there anyway to tar gzip mongo dumps like you can do with MySQL dumps?
For example, for mysqldumps, you can write a command as such:
mysqldump -u <username> --password=<password> --all-databases | gzip > all-databases.`date +%F`.gz
Is there an equivalent way to do the same for mongo dumps?
For mongo dumps I run this command:
mongodump --host localhost --out /backup
Is there a way to just pipe that to gzip? I tried, but that didn't work.
Any ideas?
Version 3.2 introduced gzip and archive option:
mongodump --db <yourdb> --gzip --archive=/path/to/archive
Then you can restore with:
mongorestore --gzip --archive=/path/to/archive
Update (July 2015):
TOOLS-675 is now marked as complete, which will allow for dumping to an archive format in 3.2 and gzip will be one of the options in the 3.2 versions of the mongodump/mongorestore tools. I will update with the relevant docs once they are live for 3.2
Original answer (3.0 and below):
You can do this with a single collection by outputting mongodump to stdout, then piping it to a compression program (gzip, bzip2) but you will only get data (no index information) and you cannot do it for a full database (multiple collections) for now. The relevant feature request for this functionality is SERVER-5190 for upvoting/watching purposes.
Here is a quick sample run through of what is possible, using bzip2 in this example:
./mongo
MongoDB shell version: 2.6.1
connecting to: test
> db.foo.find()
{ "_id" : ObjectId("53ad8a3eb74b5ae2ff0ec93a"), "a" : 1 }
{ "_id" : ObjectId("53ad8ba445be9c4f7bd018b4"), "a" : 2 }
{ "_id" : ObjectId("53ad8ba645be9c4f7bd018b5"), "a" : 3 }
{ "_id" : ObjectId("53ad8ba845be9c4f7bd018b6"), "a" : 4 }
{ "_id" : ObjectId("53ad8baa45be9c4f7bd018b7"), "a" : 5 }
>
bye
$ ./mongodump -d test -c foo -o - | bzip2 - > foo.bson.bz2
connected to: 127.0.0.1
$ bunzip2 foo.bson.bz2
$ ./bsondump foo.bson
{ "_id" : ObjectId( "53ad8a3eb74b5ae2ff0ec93a" ), "a" : 1 }
{ "_id" : ObjectId( "53ad8ba445be9c4f7bd018b4" ), "a" : 2 }
{ "_id" : ObjectId( "53ad8ba645be9c4f7bd018b5" ), "a" : 3 }
{ "_id" : ObjectId( "53ad8ba845be9c4f7bd018b6" ), "a" : 4 }
{ "_id" : ObjectId( "53ad8baa45be9c4f7bd018b7" ), "a" : 5 }
5 objects found
Compare that with a straight mongodump (you get the same foo.bson but the extra foo.metadata.json describing the indexes is not included above):
$ ./mongodump -d test -c foo -o .
connected to: 127.0.0.1
2014-06-27T16:24:20.802+0100 DATABASE: test to ./test
2014-06-27T16:24:20.802+0100 test.foo to ./test/foo.bson
2014-06-27T16:24:20.802+0100 5 documents
2014-06-27T16:24:20.802+0100 Metadata for test.foo to ./test/foo.metadata.json
$ ./bsondump test/foo.bson
{ "_id" : ObjectId( "53ad8a3eb74b5ae2ff0ec93a" ), "a" : 1 }
{ "_id" : ObjectId( "53ad8ba445be9c4f7bd018b4" ), "a" : 2 }
{ "_id" : ObjectId( "53ad8ba645be9c4f7bd018b5" ), "a" : 3 }
{ "_id" : ObjectId( "53ad8ba845be9c4f7bd018b6" ), "a" : 4 }
{ "_id" : ObjectId( "53ad8baa45be9c4f7bd018b7" ), "a" : 5 }
5 objects found
Export Mongodb as
mongodump --host <host-ip> --port 27017 --db <database> --authenticationDatabase admin --username <username> --password <password> --gzip --archive > dump_`date "+%Y-%m-%d"`.gz
Import as
mongodump --host <host-ip> --port 27017 --db <database> --authenticationDatabase admin --username <username> --password <password> --gzip --archive=mongodump.gz
If you want to do it passing uri for your MongoDB replica set cluster
Dump:
mongodump --uri='mongodb://user:pass#primary_host,secondary_host/<db-name>?replicaSet=<replica-name>&authSource=admin' --gzip --archive > dump_`date "+%Y-%m-%d"`.gz
Restore:
mongorestore --uri='mongodb://user:pass#primary_host,secondary_host/<db-name>?replicaSet=<replica-name>&authSource=admin' --gzip --archive=<dump-file>.gz