I was trying to import some data in mongodb using :
mongoimport -d blog -c posts < posts.json
But start to get this error :
2015-11-06T11:37:52.566+0530 connected to: localhost
2015-11-06T11:37:53.159+0530 error inserting documents: lost connection to server
2015-11-06T11:37:53.159+0530 Failed: lost connection to server
2015-11-06T11:37:53.159+0530 imported 0 documents
Any help would be appreciated .
try adding a -j parameter
mongoimport -d blog -c posts -j 4 < posts.json
If this does not work, try changing the j value to 8.
This post explains it:
MongoDB: mongoimport loses connection when importing big files
I also faced the same issue. I just added --batchSize 1 option. It worked like a charm.
mongoimport -d blog -c posts -j 4 < posts.json --batchSize 1
Related
I have taken over undocumented Mongo 4.4.8 cluster (PSA). I am trying to tidy it up and test thouroughly.
An original connection string:
MONGODB_URI=mongodb://${USER}:${PASS}#10.0.0.3:27017,10.0.0.6:27017,10.0.0.2:27017/bud?replicaSet=bud-replica&authSource=admin
I have enabled localhost and socket connection. I can log in from cmdline with
mongo -u ${USER} -p ${PASS}
MongoDB shell version v4.4.8
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("492e331b-417e-458a-83c7-9db6eaae0869") }
MongoDB server version: 4.4.8
I can switch db to bud and perform the queries. But if I run just
mongo
then the authentication with the same credentials does not work:
bud-replica:PRIMARY> db.auth('admin','admin');
Error: Authentication failed.
0
I tried to search for users but shows there arent any:
bud-replica:PRIMARY> db.getUsers()
[ ]
bud-replica:PRIMARY> use bud
switched to db bud
bud-replica:PRIMARY> db.getUsers()
[ ]
This is mongod.conf security part:
security:
authorization: enabled
keyFile: "/etc/bud-rs"
Finally I need to export my data before doing experiments. Though the cmd line interface looks similar, mongoexport cannot fetch the data, regardless I set user/password or skip these arguments.
mongoexport -h localhost --db=bud -u ${USER} -p ${PASS} -c=accidents --jsonArray > accidents.json
2021-08-25T19:30:30.631+0200 could not connect to server: connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.
mongoexport -h localhost --db=bud -u ${USER} -p ${PASS} -c=accidents --jsonArray --authenticationDatabase “admin” > accidents.json
2021-08-25T19:36:18.738+0200 could not connect to server: connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.
root#10:~# mongoexport -h localhost --db=bud -u ${USER} -p ${PASS} -c=accidents --jsonArray --authenticationDatabase “bud” > accidents.json
2021-08-25T19:38:21.174+0200 could not connect to server: connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.
I am really confused and I failed to find a solution on Google or SO.
Second relevant question:
If I need to create new user, shall I do it on all replicas or it is automatically synchronized?
1st update
This is the workaround, but my questions are still valid. I want to understand.
root#10:~# mongoexport --db=bud -u ${USER} -p ${PASS} -c=accidents --jsonArray "mongodb://admin:admin#10.0.0.3:27017/bud?authSource=admin" > accidents.json
2021-08-25T20:46:54.777+0200 connected to: mongodb://[**REDACTED**]#10.0.0.3:27017/bud?authSource=admin
2021-08-25T20:46:55.778+0200 [........................] bud.accidents 0/4379 (0.0%)
2021-08-25T20:46:56.497+0200 [########################] bud.accidents 4379/4379 (100.0%)
2021-08-25T20:46:56.497+0200 exported 4379 records
2nd update
bud-replica:PRIMARY> use admin
bud-replica:PRIMARY> show collections
system.keys
system.users
system.version
bud-replica:PRIMARY> db.system.users.find()
{ "_id" : "admin.admin", "userId" : UUID("769e4f5c-6f46-4153-857e-47d7d8730066"), "user" : "admin", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "32/AP4019eome36j8n
The user credential was created in the admin database.
When connecting with the mongo shell, switch with use admin before running db.auth
The mongoexport command that worked used authSource=admin in the connection string.
Add --authenticationDatabase=admin to the other command line to direct it to use the admin database for auth as well.
whole example command as below worked for me.
Mongodb version: 5.x.x, also for Mongodb version: 8.x.x
mongodump --authenticationDatabase=admin --uri mongodb://username:password#mongodb-host/db-name?ssl=false&authSource=admin
just a newbie in asking a question here. I hope I will not violate any rules with my question.
I created a Full backup (fsyncLock(), snapshot using "tar czvf", fysncUnlock()) and Incremental backup (mongodump -d local -c oplog.rs -q "(timestamp-range)") running every hour in contab (00 * * * *) by following this link below:
Mongodb incremental backups
Also full backup using the native command mongodump (fsyncLock(), mongodump host dump/, fysncUnlock()) and the same strategy using the incremental backup I mentioned above.
I can restore the full backup with no problems, for native mongodump full backup, just create a new instance and restore. For the tar file snapshot, just untar, "mongod -d mongo.cnf", and force to reconfigure (rs.reconfig(cfg, {force: true})) the replica set since upon starting it will show "rs1:OTHER>". I manage to make it "PRIMARY>" after that even only running 1 instance first.
The problem is when restoring my hourly incremental backups, it shows this error upon running the command below:
> mongorestore --port 26017 -uroot --authenticationDatabase admin
> --oplogReplay 20191204-incre/2019-12-04-00\:00\:01/ Enter password:
>
> 2019-12-04T14:23:05.456+0800 preparing collections to restore from
> 2019-12-04T14:23:05.456+0800 replaying oplog
> 2019-12-04T14:23:05.609+0800 Failed: restore error: error handling
> transaction oplog entry: error replaying transaction: error extracting
> transaction ops: applyOps field: no such field
> 2019-12-04T14:23:05.609+0800 0 document(s) restored successfully. 0
> document(s) failed to restore.
When I renamed the file "oplog.rs.bson" to "oplog.bson" and run this command instead:
> mongorestore --port 26017 -uroot --authenticationDatabase admin
> --oplogReplay --oplogFile=20191204-incre/2019-12-04-01\:00\:01/local/oplog.bson 20191204-incre/2019-12-04-01\:00\:01/ Enter password:
>
> 2019-12-04T14:10:22.739+0800 preparing collections to restore from
> 2019-12-04T14:10:22.741+0800 restoring to existing collection
> local.oplog without dropping 2019-12-04T14:10:22.741+0800 restoring
> local.oplog from 20191204-incre/2019-12-04-01:00:01/local/oplog.bson
> 2019-12-04T14:10:22.791+0800 no indexes to restore
> 2019-12-04T14:10:22.791+0800 finished restoring local.oplog (1839
> documents, 0 failures) 2019-12-04T14:10:22.791+0800 replaying oplog
>
> 2019-12-04T14:10:22.915+0800 Failed: restore error: error handling
> transaction oplog entry: error replaying transaction: error extracting
> transaction ops: applyOps field: no such field
> 2019-12-04T14:10:22.915+0800 1839 document(s) restored successfully. 0
> document(s) failed to restore.
It successfully restored the oplogs, but under "local.oplog" instead of the "local.oplog.rs", and it really didn't replay the logs on their respective database.collection.
rs1:PRIMARY> use local
rs1:PRIMARY> show tables
oplog
oplog.rs
rs1:PRIMARY> db.oplog.count()
3679
Which part did I do it wrong? Thanks!
just move your .bson file out of "local" folder. Then remove (rm) the "local" folder and continue with mongorestore.
I am trying to use mongoimport my existing .json file into the cloud cluster. But I keep getting this error
error validating settings: incompatible options: --file and positional argument(s)
I have been on the cat with mongo for an hour and they have been no help to solve my issue. I think it has to do with --file drivers.json and it finding that file.
./mongoimport --host muleapp-shard-00-02-bxny3.mongodb.net:27017 --ssl --username srusso -password MuleApp2019 --authenticationDatabase admin --db Drivers --collection driver --type json --file drivers.json
error validating settings: incompatible options: --file and positional argument(s)
now I am getting a time out error:
2019-11-08T14:12:38.533-0500 error connecting to host: could not connect to server: server selection error: server selection timeout
current topology: Type: Single
Servers:
Addr: muleapp-shard-00-02-bxny3.mongodb.net:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 3.229.7.228:27017: i/o timeout
The first error could be due to long option password being provided using - rather than --. So the option could be provided as either of --password MuleApp2019 or -p MuleApp2019.
I have some trouble importing a JSON file to a local MongoDB instance. The JSON was generated using mongoexport and looks like this. No arrays, no hardcore nesting:
{"_created":{"$date":"2015-10-20T12:46:25.000Z"},"_etag":"7fab35685eea8d8097656092961d3a9cfe46ffbc","_id":{"$oid":"562637a14e0c9836e0821a5e"},"_updated":{"$date":"2015-10-20T12:46:25.000Z"},"body":"base64 encoded string","sender":"mail#mail.com","type":"answer"}
{"_created":{"$date":"2015-10-20T12:46:25.000Z"},"_etag":"7fab35685eea8d8097656092961d3a9cfe46ffbc","_id":{"$oid":"562637a14e0c9836e0821a5e"},"_updated":{"$date":"2015-10-20T12:46:25.000Z"},"body":"base64 encoded string","sender":"mail#mail.com","type":"answer"}
If I import a 9MB file with ~300 rows, there is no problem:
[stekhn latest]$ mongoimport -d mietscraping -c mails mails-small.json
2015-11-02T10:03:11.353+0100 connected to: localhost
2015-11-02T10:03:11.372+0100 imported 240 documents
But if try to import a 32MB file with ~1300 rows, the import fails:
[stekhn latest]$ mongoimport -d mietscraping -c mails mails.json
2015-11-02T10:05:25.228+0100 connected to: localhost
2015-11-02T10:05:25.735+0100 error inserting documents: lost connection to server
2015-11-02T10:05:25.735+0100 Failed: lost connection to server
2015-11-02T10:05:25.735+0100 imported 0 documents
Here is the log:
2015-11-02T11:53:04.146+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45237 #21 (6 connections now open)
2015-11-02T11:53:04.532+0100 I - [conn21] Assertion: 10334:BSONObj size: 23592351 (0x167FD9F) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "mails"
2015-11-02T11:53:04.536+0100 I NETWORK [conn21] AssertionException handling request, closing client connection: 10334 BSONObj size: 23592351 (0x167FD9F) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "mails"
I've heard about the 16MB limit for BSON documents before, but since no row in my JSON file is bigger than 16MB, this shouldn't be a problem, right? When I do the exact same (32MB) import one my local computer, everything works fine.
Any ideas what could cause this weird behaviour?
I guess the problem is about performance, any way you can solved used:
you can use mongoimport option -j. Try increment if not work with 4. i.e, 4,8,16, depend of the number of core you have in your cpu.
mongoimport --help
-j, --numInsertionWorkers= number of insert operations to run
concurrently (defaults to 1)
mongoimport -d mietscraping -c mails -j 4 < mails.json
or you can split the file and import all files.
I hope this help you.
looking a little more, is a bug in some version
https://jira.mongodb.org/browse/TOOLS-939
here another solution you can change the batchSize, for default is 10000, reduce the value and test:
mongoimport -d mietscraping -c mails < mails.json --batchSize 1
Quite old, but I struggled on same issue.
If you want to import big files, especially remote with Compass or by Program just add
&wtimeoutMS=0
to your Connection-String. This removes Timeout on Write-Operations.
I'm trying to dump a database from another server (this works fine), then restore it on a new server (this does not work fine).
I first run:
mongodump --host -d
This creates a folder dump/db which contains all of the bson documents.
Then in the dump folder, I'm running:
mongorestore -d dbname db
This works and iterates through the files, but I get this error on dbname.system.users
Wed May 23 02:08:05 { key: { _id: 1 }, ns: "dbname.system.users", name: "_id_" }
Error creating index dbname.system.usersassertion: 13111 field not found, expected type 16
Any ideas how to resolve this?
If it realy different versions, use --noIndexRestore option. And create all index after that.
Any chance the source and destination are different versions?
In any case, to get around this, restore the collections individually using the -c flag to the target DB and then build the indexes afterward. The system collection is the one used for indexes, so it is fairly easy to recreate - try it last once everything else has been restore, and if it still fails you can always just recreate the relevant indexes.
The issue could also caused by this bug in older versions of Mongo (In my case it was 2.0.8):
https://jira.mongodb.org/browse/SERVER-7181
Basically, you get 13111 field not found, expected type 16 error when it should actually be prompting you to enter your authentication details.
And example of how I fixed it:
root#precise64:/# mongorestore /backups/demand/ondemand.05-24-2013T114223/
connected to: 127.0.0.1
[REDACTED]
Fri May 24 11:48:15 going into namespace [test.system.indexes]
Fri May 24 11:48:15 { key: { _id: 1 }, ns: "test.system.users", name: "_id_" }
Error creating index test.system.usersassertion: 13111 field not found, expected type 16
# Error when not giving username and password
root#precise64:/# mongorestore -u fakeuser -p fakepassword /backups/demand/ondemand.05-24-2013T114223/
connected to: 127.0.0.1
[REDACTED]
Fri May 24 11:57:11 /backups/demand/ondemand.05-24-2013T114223/test/system.users.bson
Fri May 24 11:57:11 going into namespace [test.system.users]
1 objects found
# Works fine when giving username and password! :)
Hope that helps anyone who's issue doesn't get fixed by the previous 2 replies!
This can also happen if you are trying to mongorestore into MongoDB 2.6+ and the dump you are trying to restore contains a system.users table in any database other than admin. In MongoDB 2.2 and 2.4 the system.userscollections could occur in any database. The auth schema migration associated with MongoDB 2.6 moved all users into the system.users table in the admin database, but left behind the system.users tables in the other databases (MongoDB 2.6 just ignores these). This seems to cause this assertion when importing into MongoDB 2.6.