New to graph databases and trying to learn a few platforms for my senior project. I'm working on ArangoDB and can't seem to import data correctly.
The manual said to run the arangoimp file to execute import commands, but every time I launch the arangoimp file, it asks for a password (which is blank) and then exits with the following...
/Applications/ArangoDB3-CLI.app/Contents/Resources/arangoimp ; exit;
Please specify a password:
Connected to ArangoDB 'http+tcp://127.0.0.1:8529',
version 3.3.15, database: '_system', username: 'root'
----------------------------------------
database: _system
collection:
create: no
source filename:
file type: json
threads: 2
connect timeout: 5
request timeout: 1200
----------------------------------------
2018-09-20T19:17:30Z [25483] FATAL Collection name is missing.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
Total noob here, so help is GREATLY appreciated... Working on a mac...
arangoimp is a command line tool that needs a few arguments. E.g. the collection you want to fill and the source data.
So you should use something like:
arangoimp --file "mydatafile.json" --type json --collection "mycollection"
Read the arangoimp documentation for more detailed information about all options and access to remote databases.
Related
I am following the https://deepstream.io/tutorials/plugins/database/postgres/
I have create a testDB with owner and permission to user james
In the conf/config.yml, I have the following configuration
plugins:
storage:
name: postgres
options:
user: james
database: testDB
password: james
host: localhost
port: 5432 #postgres default post
schema: ds #schema defaults to ds. Will be created if it doesn't exist
max: 10 #concurrent connections
idleTimeoutMillis: 30000 #timeout after which connection will be cut
writeInterval: 200 #amout of milliseconds during which writes will be buffered
notifications:
CREATE_TABLE: true #Get notified when tables are created
DESTROY_TABLE: true #Get notified when tables are dropped
INSERT: true # Get notified when records are created
UPDATE: false # Get notified when records are updated
However, when I run deepstream start, I got the following error
CONFIG_TRANSFORM | Loaded content from /Users/james/Workspace/deepstream.io/conf/permissions.yml for fileLoad(permissions.yml)
CONFIG_ERROR | Error loading module, exiting
Could someone please tell me what I have missed. Thank you for your help.
regards,
Johan
finally, I solved the problem, the configuration is correct, just to make sure that the postgres database and the role is done properly. The role needs to have a password and also LOGIN. Also, I clone the codes from github, so I can see what the error is when npm start. This will shows you verbose error message. If no error shown, npm test, this will show you all the missing packages and try to npm install again
I recently started working with Robot framework. So I had a requirement where I needed to connect with Postgres db.
So though I am able to connect with the db but then when I try to execute queries, the flow is getting stuck. Even the test is not failing. Following is what I did:
Connect To Database psycopg2 ${DBName} ${DBUser} ${DBPass} ${DBHost} ${DBPort}
${current_row_count} = Row Count Select * from xyz
The first statement is executing fine but then it gets stuck on second statement.
Can somebody help me out on this
To Execute Query and get data from result :
Connect To Database psycopg2 ${DBName} ${DBUser} ${DBPass} ${DBHost} ${DBPort}
${output} = Query SELECT * from xyz;
Log ${output}
${DataResults}= Get from list ${output} 0
${DataResults}= Convert to list ${DataResults}
${DataResults}= Get from list ${DataResults} 0
${DataResults} convert to string ${DataResults}
Disconnect From Database
You are not executing your query.... read below a bit documentation and an example ;)
In the example you can see example variable but introduce your data ;)
Name: Connect To Database Using Custom Params
Source: DatabaseLibrary
Arguments:
[ dbapiModuleName=None | db_connect_string= ]
Loads the DB API 2.0 module given dbapiModuleName then uses it to connect to the database using the map string db_custom_param_string.
Example usage Example usage: :
Connect To Database Using Custom Params pymssql database='${db_database}' , user='${db_user}', password='${db_password}', host='${db_host}'
${queryResults} Query ${query}
Disconnect From Database
I have some trouble importing a JSON file to a local MongoDB instance. The JSON was generated using mongoexport and looks like this. No arrays, no hardcore nesting:
{"_created":{"$date":"2015-10-20T12:46:25.000Z"},"_etag":"7fab35685eea8d8097656092961d3a9cfe46ffbc","_id":{"$oid":"562637a14e0c9836e0821a5e"},"_updated":{"$date":"2015-10-20T12:46:25.000Z"},"body":"base64 encoded string","sender":"mail#mail.com","type":"answer"}
{"_created":{"$date":"2015-10-20T12:46:25.000Z"},"_etag":"7fab35685eea8d8097656092961d3a9cfe46ffbc","_id":{"$oid":"562637a14e0c9836e0821a5e"},"_updated":{"$date":"2015-10-20T12:46:25.000Z"},"body":"base64 encoded string","sender":"mail#mail.com","type":"answer"}
If I import a 9MB file with ~300 rows, there is no problem:
[stekhn latest]$ mongoimport -d mietscraping -c mails mails-small.json
2015-11-02T10:03:11.353+0100 connected to: localhost
2015-11-02T10:03:11.372+0100 imported 240 documents
But if try to import a 32MB file with ~1300 rows, the import fails:
[stekhn latest]$ mongoimport -d mietscraping -c mails mails.json
2015-11-02T10:05:25.228+0100 connected to: localhost
2015-11-02T10:05:25.735+0100 error inserting documents: lost connection to server
2015-11-02T10:05:25.735+0100 Failed: lost connection to server
2015-11-02T10:05:25.735+0100 imported 0 documents
Here is the log:
2015-11-02T11:53:04.146+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:45237 #21 (6 connections now open)
2015-11-02T11:53:04.532+0100 I - [conn21] Assertion: 10334:BSONObj size: 23592351 (0x167FD9F) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "mails"
2015-11-02T11:53:04.536+0100 I NETWORK [conn21] AssertionException handling request, closing client connection: 10334 BSONObj size: 23592351 (0x167FD9F) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "mails"
I've heard about the 16MB limit for BSON documents before, but since no row in my JSON file is bigger than 16MB, this shouldn't be a problem, right? When I do the exact same (32MB) import one my local computer, everything works fine.
Any ideas what could cause this weird behaviour?
I guess the problem is about performance, any way you can solved used:
you can use mongoimport option -j. Try increment if not work with 4. i.e, 4,8,16, depend of the number of core you have in your cpu.
mongoimport --help
-j, --numInsertionWorkers= number of insert operations to run
concurrently (defaults to 1)
mongoimport -d mietscraping -c mails -j 4 < mails.json
or you can split the file and import all files.
I hope this help you.
looking a little more, is a bug in some version
https://jira.mongodb.org/browse/TOOLS-939
here another solution you can change the batchSize, for default is 10000, reduce the value and test:
mongoimport -d mietscraping -c mails < mails.json --batchSize 1
Quite old, but I struggled on same issue.
If you want to import big files, especially remote with Compass or by Program just add
&wtimeoutMS=0
to your Connection-String. This removes Timeout on Write-Operations.
After mongodump, I did mongorestore which seemed to work fine
heathers-air:db heathercohen$ mongorestore -v -host localhost:27017
2015-02-06T11:22:40.027-0800 creating new connection to:localhost:27017
2015-02-06T11:22:40.028-0800 [ConnectBG] BackgroundJob starting: ConnectBG
2015-02-06T11:22:40.028-0800 connected to server localhost:27017 (127.0.0.1)
2015-02-06T11:22:40.028-0800 connected connection!
connected to: localhost:27017
2015-02-06T11:22:40.030-0800 dump/langs.bson
2015-02-06T11:22:40.030-0800 going into namespace [dump.langs]
Restoring to dump.langs without dropping. Restored data will be inserted without raising errors; check your server log
file dump/langs.bson empty, skipping
2015-02-06T11:22:40.030-0800 Creating index: { key: { _id: 1 }, name: "_id_", ns: "dump.langs" }
2015-02-06T11:22:40.031-0800 dump/tweets.bson
2015-02-06T11:22:40.031-0800 going into namespace [dump.tweets]
Restoring to dump.tweets without dropping. Restored data will be inserted without raising errors; check your server log
file size: 4877899
30597 objects found
2015-02-06T11:22:41.883-0800 Creating index: { key: { _id: 1 }, name: "_id_", ns: "dump.tweets" }
When I try to access the data though, it's still empty and the way it looked before restore:
> show dbs
admin (empty)
dump 0.078GB
local 0.078GB
tweets (empty)
twitter (empty)
It says it found 30597 objects, where did they go?
They went into the dump database, and then into the collections dump.tweets and dump.langs. The fact that the files are contained in the folder dump means that mongorestore thinks they should be restored to the database dump (it is inferred from the path). The verbose output even explicitly states that the data is being placed into dump.langs and dump.tweets specifically.
If you specify the database you wish to restore to (with -d) and restore the specific files you will be able to restore the documents to the database you desire. Or, you can simply have a look in the dump database by running:
use dump;
db.tweets.find();
db.langs.find();
Created mysqldump file and uploaded to bucket: zip_zata_mysqldump and even though import does not show any failure, I don't see the databases created in the db.
How I can see what went wrong so I can fix?
If import failed the operations tab shows a generic error "An unknown error occurred".
To receive a detailed error you can execute the import from command line with the following syntax:
mysql -h MYSQL_INSTANCE_IP -u USER -p DATABASE_NAME < SQL_FILE_TO_IMPORT
For your reference import and export features are documented at [1].
Regards,
Paolo
Link:
[1] - https://cloud.google.com/sql/docs/import-export
For us adding --hex-blob in the mysqldump command allowed us to import. It failed An unknown error occurred without that flag.