I have a large dump (millions of nodes and relationships) from a Neo4J 2.2.5 database in Cypher format (produced with neo4j-sh -c dump), that I'm trying to import into a 3.0.3 instance.
However, the import process (neo4j-sh < dump.cypher) slows down drastically after a few minutes, down to a couple records per second.
Is there any way to speed up this process? I tried upgrading the database as described in the manual, but the new instance crashes with an exception about a version mismatch in the store format.
Neo4j 3.0 comes with a bin/neo4j-admin tool for exactly this purpose.
try bin/neo4j-admin import --mode database --from /path/to/db
see: http://neo4j.com/docs/operations-manual/current/deployment/upgrade/#upgrade-instructions
The cypher dump is not useful for large database, it's only for smaller setups (a few thousand nodes) for demos etc.
FYI: In Neo4j 3.0 the cypher export procedure from APOC is much more suited for large scale cypher dumps.
Update
You can also try to upgrade from 2.2 to 2.3 first. E.g by using neo4j-shell
add allow_store_upgrade=true to your neo4j.properties` in 2.3
and then do: bin/neo4j-shell -path /path/to/db -config conf/neo4j.properties -c quit
If it is finished that backup of your db is on Version 2.3
Then you should be able to use neo4j-admin -import ...
I recently had this same symptom with my CSV import slowing to death.
My load-csv cypher script had too many rels.
So I divided my load in two. First create the nodes, then the relations and most connected nodes. HIH.
Back to your issue
First, try to increase the memory for the JVM. In NEO/conf, there is a wrapper file. At the beginning are the memory settings.
Lastly, from an instance w/ your data, export to multiple CSV files and import them in your new server.
Related
I'd like to be able to upgrade my existing cloudsql postgres 9.6 instance to 11 to use some new pg 11 features.
I've been trying to figure out a good migration plan but it seems like the only option available is sql dump and restore. The database is 100Gig+ so this will take quite some time, and I'd like to avoid downtime as much as possible. Are there any options available? I was considering enabling statement logging: log_statement=mod, creating a dump, importing it into a pg-11 instance taking down the db + then scraping the logs to reply the latest updates into the pg-11 instance by downloading the logs and writing a script to re-run the inserts. Seems doable, but doesn't feel nice.
I am wondering if anyone faced this before and has had any other solutions?
Postgres 11 on Cloud SQL is still in Beta. It is not recommended to be using a product that is in Beta on a production environment.
However, should you choose to proceed, you must export the data by either creating a SQL dump or putting the data into a .csv file (depending on your needs)(best practices) create a Postgres 11 instance, and then import the data.
For the data that won’t be in the dump, you can either:
a) Do what you have suggested by logging the queries and then re-run the inserts
b) Create a dump, import it onto the new instance make it live and then take another dump of the old one again, compare to remove duplicates and import the differences. This will be difficult if you have auto-incrementing primary keys.
c) Create the schema on the Postgres 11 instance and deploy it. Then create the dump and import at a later time. If you have primary keys as auto incrementing, alter the schema to start at a value that you would like.
I am trying to dump the schema and data from an existing Oracle DB and import it into another Oracle DB.
I have tried using the "Export Wizard" provided by sqldeveloper.
I found answers using Oracle Data Pump, however i do not have access to the filesystem of the DB server.
I expect to get a file that i can copy and import into another DB
Without Data Pump, you have to make some concessions.
The biggest concession is you're going to ask a Client application, running somewhere on your network, to deal with a potentially HUGE amount of data/IO.
Withing reasonable limits, you can use the Tools > Database Export wizard to build a series of SQLPlus style scripts, both DDL (CREATEs) and DATA (INSERTs).
Once you have those scripts, you can use SQLPlus, SQLcl, or SQL Developer to run them on your new/target database.
While restoring a (pg_dump-produced) database dump, I get the following error:
Cannot execute COPY FROM on a distributed table on master node
How can I work around this?
COPY support was added in Citus 5.1, which was released May 2016 and is available in the official PostgreSQL Linux package repositories (PGDG).
Are you trying to load data via a pg_dump output? Creating distributed tables is slightly different than regular tables, and requires picking of partition columns and partitioning method. Take a look at the docs to get more information on both.
We have one application where we have db2 10.1 as database.
Now one requirement came in which we need to interface few tables to HOST which is on IBM DB2 VSE 7.4
I tried to execute load command with client option but it give "SQL1325N The remote database environment does not support the command or one
of the command options." error.
command is :"D:\tempdata>db2 load client from app.tbl of ixf
insert into host.tbl"
Many post says that its not allow to use load from 10.1 to VSE Z/OS.
Another option I tried is import but its too slow and we need to delete records every time as truncate is not available.
Replication can be think for option but we would like to avoid replication.
Can anyone suggest way to achieve this. Load can be use or not?
It seems its not allow to use load from remote machine. But wondering what is use of CLIENT option in load then.
Finally we decided to use import utility after deleting HOST DB2 records. In that we need to execute delete and import command on part of table. If we try to import or delete big table at one go it give error of log file size full.
Hope this will help.
I have one live website with multiple active users(around 30K) and each of them have their own configuration to render there homepages. The current stack of the portal is Java Spring Hibernate with SQL Server. Now, we have re written the code in Python MongoDB stack and want to migrate our users to new system. The issue here is that the old and new code will be deployed on the separate machines and we want to run this migration for few users as part of Beta Testing. Once the Beta testing is done, we will migrate all the users.
What would be the best approach to achieve this? We are thinking about dumping the data in alternative file system like XML/JSON on a remote server and then reading it in the new code.
Please suggest what should be the best way to accomplish this task
Import CSV, TSV or JSON data into MongoDB.
It will be faster and optimal to dump the file in a format like json, txt or csv , which should be copied to the new server and then import the data using mongoimport, in the command line shell.
Example
mongoimport -d databasename -c collectionname < users.json
Kindly regard to the link below for more information on mongoimport if you need to
http://docs.mongodb.org/manual/reference/mongoimport/