I am either misusing mongodump or it has a bug, but I'm not sure which. I typically use mongo connection strings in my applications and scripts, e.g.
mongo mongodb://username:ps#myhostname/dbname this works
The mongodump tool supposedly supports URL strings, but every time I try to use it it starts and then does nothing:
mongodump --uri mongodb://username:ps#myhostname/dbname this runs but stops and does nothing with no CPU usage.
I've tried using -vvvvv and there is no interesting data shown.
If I do the exact same thing using the "old" parameters, it works, but then I'd have to parse URIs and that makes me sad:
mongodump --host myhostname --username username --password ps -d dbname this works
1) Am I doing this wrong?
2) If this is a bug, where would I file a ticket?
3) Is there a tool that would parse a mongodb:// URI back into pieces so that I can keep using URIs in my automation stack?
$ mongodump --version
mongodump version: r3.6.8
git version: 6bc9ed599c3fa164703346a22bad17e33fa913e4
Go version: go1.8.5
os: linux
arch: amd64
compiler: gc
OpenSSL version: OpenSSL 1.1.0f 25 May 2017
db.version() in a connected shell also returns 3.6.8
I ran into this same issue, and likewise, was quite sad. However, I'm happy again because I realized you MUST append the following two options to your connection string:
?ssl=true&authSource=admin
Pop those bad boys on your uri and you should be smooth sailing.
Related
I am trying to create a mongodump of my atlas database, but I get the following error:
error parsing command line options: error parsing uri (mongodb+srv://blablabla:123#blablabla.blabla.blala.mongodb.net/DATABASENAME): scheme must be "mongodb"
try 'mongodump --help' for more information
Any idea, I am trying to automate my backups using a bash script.
I tried to change the url from mongodb+srv to mongodb but nothing happen
You are using an old version of mongodump that doesn't recognize mongodb+srv URIs. Upgrade mongodump or use a non-SRV URI.
I am using a command like this to dump data from a remote machine:
mongodump --verbose \
--uri="mongodb://mongousr:somepassword#host.domain.com:27017/somedb?authSource=admin" \
--out="$BACKUP_PATH"
This fails like so:
Failed: error writing data for collection `somedb.someCollection` to disk: error reading collection: EOF
somedb.someCollection is about 40GB. I don't have the ability to increase RAM to this size.
I have seen two explanations. One is that the console output is too verbose and fills the RAM. This seems absurd, it's only a few kilobytes and it's on the client machine anyway. Rejected (but I am trying it again now with --quiet just to be sure).
The more plausible explanation is that the host fills its RAM with somedb.someCollection data and then fails. The problem is that the 'solution' that I've seen proposed is to increase the RAM to be bigger than the size of the collection.
Really? That can't be right. What's the point of mongodump with that limitation?
The question: is it possible to mongodump a database with a collection that is larger than my RAM size? How?
mongodump Client:
macOS
mongodump --version
mongodump version: 4.0.3
git version: homebrew
Go version: go1.11.4
os: darwin
arch: amd64
compiler: gc
OpenSSL version: OpenSSL 1.0.2r 26 Feb 2019
Server:
built with docker FROM mongo:
Reports: MongoDB server version: 4.0.8
Simply dump your collection slice by slice:
mongodump --verbose \
--uri="mongodb://mongousr:somepassword#host.domain.com:27017/somedb?authSource=admin" \
--out="$BACKUP_PATH" -q '{_id: {$gte: ObjectId("40ad7bce1a3e827d690385ec")}}'
mongodump --verbose \
--uri="mongodb://mongousr:somepassword#host.domain.com:27017/somedb?authSource=admin" \
--out="$BACKUP_PATH" -q '{_id: {$lt: ObjectId("40ad7bce1a3e827d690385ec")}}'
or partitioning your dump by a different query set on _id or some different field. The reported _id is a mere example.
Stennie's answer really works.
The default value of storage.wiredTiger.engineConfig.cacheSizeGB is max((RAM-1GB)/2, 256MB). If your mongodb server is running in a docker container with default configs, and there are other apps running in the host machine, the memory could be full filled when you are dumping a large collection. The same thing can happen if the containers' RAM is limited due to your configs.
You can use docker run --name some-mongo -d mongo --wiredTigerCacheSizeGB 1.5 (the number is based on you situation).
Another possibility is to add the compress flag to the output of mongodump. It helped me to backup a db that hanged at 48% without compressing. So the syntax would be:
mongodump --uri="mongodb://mongousr:somepassword#host.domain.com:27017/somedbauthSource=admin" --gzip --out="$BACKUP_PATH"
Since I use mongodb-clients 2.6.10 the mongodump doesn't work anymore. With the previous version 3.4.7 everything worked fine. It is a dedicated mongodb database as a service in the CF AppCloud where nothing has been changed. Unfortunately, it is not possible to use version 3.4.7 again.
Does anyone have an idea why it doesn't work anymore?
vcap#host:~$ mongodump -u XXX -p XXX -d XXX --authenticationDatabase XXX -h kubernetes-service-node.service.consul:XXX,kubernetes-service-node.service.consul:XXX,kubernetes-service-node.service.consul:XXX
Result: https://jsfiddle.net/yz1kp68p/
Judging from the error, it's probably got nothing to do with the mongodump version. Can you generally connect to the database (i.e. with the mongo shell instead of mongodump)? My guess is that the app either isn't bound (cf bind-service) to the database or hasn't been restaged (cf restage) after being bound - both is necessary to enable firewall access from the app to the database. Also, why can't you use a newer mongodump version anymore? Sounds more like that's what needs to be addressed in the first place.
I successfully installed mongo-tools from the Ubuntu artful repository to have a mongodump version that supports SCRAM-SHA-1 authentication mechanism. The dumper app now works without problems.
Installing mongodb-clients out of the artful repository did not work in my case, but mongo-tools did it.
The mongo client can connect with a standard URI:
mongo mongodb://<dbuser>:<dbpassword>#<server>:<port>/<db>
However, mongodump seems to require an awkward syntax breaking this up into different arguments:
mongodump -u dbuser -p dbpassword -h server -p port -d db ...
Is there a quick and easy way to pass a URI to mongodump as well?
The --uri option was added within a minor release of MongoDB 3.4.6. This was referenced in the JIRA issue TOOLS-1587.
It actually did not get official documentation until the MongoDB 3.6 release, but it's now in the manual page
--uri
New in version 3.4.6.
Specify a resolvable URI connection string for the mongod to which to connect.
The following is the standard URI connection scheme:
mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
For detailed explanations of the components of this string, refer to the Connection String URI Format documentation.
The same --uri option is added to other tools such as mongoexport, mongoimport and mongorestore.
Answer: 05-15-2020
I know this is a late answer, but it solved my problem.
To dump using an URI, do:
mongodump --uri=[uri]/[db]
where:
uri is your URI (e.g. mongodb+srv://john:xxxxxxxxxxxxxxx#cluster0-jdtjt.mongodb.net)
db is the database that you want to dump (e.g. sales)
So in this artificial example, it would be like this:
mongodump --uri=mongodb+srv://john:xxxxxxxxxxxxxxx#cluster0-jdtjt.mongodb.net/sales
So if this really did not work, edit the file /etc/resolv.conf and change nameserver value to e.g. 8.8.8.8, like this:
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
# nameserver 127.0.0.53
nameserver 8.8.8.8
options edns0
Now it should works.
Easy steps to getting dump and restore whole database at once with any data loses
take dump in the local machine
mongodump --uri "mongodb+srv://username:password#dbexmplae-fsddf.igt.mongodb.net/dbname" --out "C:\databaseDump"
**locate to the dumped folder and restore to local database **
mongorestore --db dbname C:\databaseDump\dbname
restore to remote database
mongorestore --uri "mongodb+srv://username:password#dbexmplae-fsddf.igt.mongodb.net/dbname" --db dbname C:\databaseDump\dbname
A simple and easy string to understand
mongodump --uri='mongodb://...url/...db-name' --out $(pwd)
or
mongodump --uri='mongodb+srv://...url/...db-name' --out $(pwd)
Depending on your implementation, will dump the whole database of $(db-name) located at $(url) and place the contents into your current working directory $(pwd)
Even if you supply the uri option, you still need to supply the db option. The documentation at https://docs.mongodb.com/manual/reference/program/mongorestore/ is incorrect when it says that the db option is "incompatible" with the uri option. If you don't supply the db option together with the uri option you get a don't know what to do with file error for each .bson file that you're trying to restore. I tested using mongorestore version 3.6.9 on ubuntu and from mac osx.
Basically, I have a problem with using mongodump to back up my MongoDB.
This is the general syntax I use in SSH:
mongodump -d myDatabaseName -o ~/backups/my_backup
This is the resulting message:
Fri Apr 22 20:39:57.304 DATABASE: myDatabaseName to /root/backups/my_backup/myDatabaseName
This simply generates a blank folder with no files in it whatsoever. The actual database is fairly large, so not sure what's going on.
I would also like to add that my mongodump client and my MongoDB version are both the same (version 2.4.9).
Not sure how to go about fixing this. Any help is appreciated.
This is a similar question as Mongodump getting blank folders
There is no accepted answer as of writing my answer. Here is what I did to resolve my issue and I believe it will help you as well.
The default mongodb-client deb package with Ubuntu is the issue. I removed those and installed the mongodb-org-tools package from mongodb.com https://docs.mongodb.com/master/tutorial/install-mongodb-on-ubuntu/?_ga=2.36209632.1945690590.1499275806-1594815486.1499275806
They have other install instructions for your specific OS if you are not on Ubuntu https://www.mongodb.com/download-center?jmp=nav#community
Try adding the port of mongodb_port as in:
mongodump --port your_number -c the_collection -d the_database
Make sure that you have the exact name of the database. If you spell it wrong, this could happen. To confirm, connect to your mongo database and type show dbs to see a list of database names. Then make sure that your databasename parameter -d <databasename> matches one of those in the list.