I have a dev server with a copy of a database which can be edited and a live real server with that same database in a different state. To move the database from the dev to the live I run from the dev server:
mysqldump -u root -p --opt db_name tbl_name | mysql -u user_name -p --host=live_IP -C db_name
With the appropriate values in db_name, tbl_name, user_name, and live_IP. However, this currently drops the table on the live server and copies the dev version over -- effectively overwriting everything and trashing any new data in the live table. What I really want is to have new lines in the dev server and conflicting lines come from the dev server's copy but that any new lines in the live server's copy to remain unchanged. This is some sort of a merge and I have not been able to find any mention to something like this in the documentation but it seems like it should be possible since it is a common need.
You can do this by adding mysqldump options before the pipe. In your case I think you want --insert-ignore, --no-create-db, and --no-create-info.
Related
I need to copy a schema in Postgres to another database on a remote server, but I keep up ending to get a fail like:
pg_dump: too many command-line arguments (first is "--n")
My code:
pg_dump postgres -n my_local_shema | psql -h 11.22.33.44 -U my_user_on_remote_server-d postgres
I have tried for hours and with different commands but I keep getting the "too many command lines".
Try with a reversed order like this:
pg_dump -n my_local_shema postgres
Ok. This command structure works as a charm:
pg_dump -n my_local_shema_name -d my_local_database -U my_local_username | psql -h 111.222.333.444 -U my_user_name_on_remote_Server my_Database_name_on_remote_server
Step-by-step-guide
I copied a shema with all tables and indexes to another database on another server.
The 111.222.333.444 is the IP of the remote server.
In preparation(I dont know if it is actually needed), I first created a shema on the remote server with an identical shemaname as the one, I wanted to copy. I also checked, that the firewall was open for datatransfer from the old server to the new one.
Then, i opened a commandpromt (I use windows) and opened the folder where the pgdump.exefile was. Here typed the command.
Last it asked me to type in a password. First it promted it. THen it was silent - nothing happened, and I did not know, what to expect. Last I typed in the password 2 times (i use the same password both on the old server and the new, upgraded one). Then things started to work and it wrote a lot with alter table, ect, ect.
Hope others can use it. :-)
I have a site that uses PostgreSQL. All content that I provide in my site is created at a development environment (this happens because it's webcrawler content). The only information created at the production environment is information about the users.
I need to find a good way to update data stored at production. May I restore to production only the tables updated at development environment and PostgreSQL will update this records at production or the best way would be to backup the users information at production, insert them at development and restore the whole database at production?
Thank you
You can use pg_dump to export the data just from the non-user tables in the development environment and pg_restore to bring that into prod.
The -t switch will let you pick specific tables.
pg_dump -d <database_name> -t <table_name>
https://www.postgresql.org/docs/current/static/app-pgdump.html
There are many tips arounds this subject here and here.
I'd suggest you to take a look on these links before everything.
If your data is discarded at each update process then a plain dump will be enough. You can redirect pg_dump output directly to psql connected on production to avoid pg_restore step, something like below:
#Of course you must drop tables to load it again
#so it'll be reasonable to make a full backup before this
pg_dump -Fp -U user -h host_to_dev -T=user your_db | psql -U user -h host_to_production your_db
You might asking yourself "Why he's saying to drop my tables"?
Bulk loading data on a fresh table is faster than deleting old data and inserting again. A quote from the docs:
Creating an index on pre-existing data is quicker than updating it incrementally as each row is loaded.
Ps¹: If you can't connect on both environment at same time then you need to do pg_restore manually.
Ps²: I don't recommend it but you can append --clean option on pg_dump to generate DROP statements automatically. Be extreme careful with this option to avoid dropping unnexpected objects.
I have two apps, with same tables. One of app collecting data from web. I want to send the datas to my second(web app)'s app database.
With the code below, I have created the file with datas:
pg_dump -U username -t public."table_name" -d database name --inserts > table_name.sql
The problem is that I just want to insert data's which does not exist in second database.
If I try the code below, I get a lot of already exists errors:
psql -U username second_database_name < table_name.sql
One of error:
multiple primary keys for table "table_name" are not allowed
Another one:
relation "table_name_attribute_442....c74_uniq" already exists
--clean , --if-exists ... What should I do?
The way I did it, was to do a pg_dump that creates a compressed archive suitable for use with pg_restore, which has the needed flags to allow the data to be imported without throwing errors.
For example:
pg_dump -Fc -h 127.0.0.1 {db_name_here} > {dump_file_name_here}
The "-Fc" gets you the file-type that pg_restore wants; it will reject a dump made without those magic letters.
Now you can restore the file with:
pg_restore -O -h 127.0.0.1 --clean --disable-triggers -d {target_db_name} {dump_file_name_here}
Voila - the data is now in the target_db.
If the 'psql' command has the equivalent flags, I don't know what they are, and did not find them in SO searching. But, hopefully, this provides a DB dump/restore that 'just works' for those who need to get back to using the DB, instead of fiddling with trying to get a simple dump/restore to happen with the expected behavior.
Also note, if you do not specify ...
-h 127.0.0.1
... it will go looking for a unix-socket, which may or may not be configured correctly. Chances are, your use-case for the DB addresses the db that way, so you never manually-configured the unix-socket to match whatever the command defaults to trying to find, which is different than how setup sets the default socket (of course, because things "just working," so you can "just use it" is just not possible).
When developing, I need to pull the latest database so I know I'm working with the latest data. However, we keep a table full of Archives that I don't need to bother downloading because it's a very large table.
I know pg_dump allows for custom parameters that will let you exclude a certain table from being dumped.
Without doing anything crazy like having 2 databases, 1 for data and 1 for archives, is there any way to download everything BUT the archives table from Heroku?
I still need it to keep backups of the archives table, but I don't want to be downloading it. Can I just do a pg_dump when needed that is seperate from the backups?
I know it's a long shot, but any suggestions would be greatly appreciated.
You can't add any custom pg_dump options when using heroku pg:backups capture. This command actually calls an undocumented Heroku Postgres API and it doesn't pass any parameters (see here for the code if you are curious).
What you can do is run your own pg_dump dump command that points to the Heroku Postgres instance.
Get the connection info with pg:credentials where DATABASE_URL can also be the the database color if you have more than one database attached to the app:
> heroku pg:credentials DATABASE_URL --app app_name
Connection info string:
"dbname=zzxcasdqwe host=ec2-1-1-1-1.compute-1.amazonaws.com port=1111 user=asdfasdf password=qwertyqwerty sslmode=require"
Connection URL:
postgres://asdfasdf:qwertyqwerty#ec2-1-1-1-1.compute-1.amazonaws.com:1111/zzxcasdqwe
Take either the the connection info string or the connection url and include that as the first argument to pg_dump and add your custom options
pg_dump "dbname=zzxcasdqwe host=ec2-1-1-1-1.compute-1.amazonaws.com port=1111 user=asdfasdf password=qwertyqwerty sslmode=require"\
-n schema -t table -O -x -Fc -f dump.out
# OR
pg_dump postgres://asdfasdf:qwertyqwerty#ec2-1-1-1-1.compute-1.amazonaws.com:1111/zzxcasdqwe \
-n schema -t table -O -x -Fc -f dump.out
I also co-wrote a Heroku plugin (parse_db_url) that will parse DATABASE_URL's into other formats like pg_dump, pg_restore, pgpass etc. I find it useful when dealing with several different Heroku databases.
I have a database containing a very large table including binary data which I want to update on a remote machine, once a day. Rather than dumping the entire table, transferring and recreating it on the remote machine, I want to dump only the updates, then use that dump to update the remote machine.
I already understand that I can produce the dump file as such.
mysqldump -u user -p=pass --quick --result-file=dump_file \
--where "Updated >= one_day_ago" \
database_name table_name
1) Does the resulting "restore" on the remote machine
mysql -u user -p=pass database_name < dump_file
only update the necessary rows? Or are there other adverse effects?
2) Is this the best way to do this? (I am unable to pipe to the server directly and using --host option)
If you only dump rows where the WHERE clause is true, then you will only get a .sql file that contains the values you want to update. So you will never get duplicate values if you use the current export options. However, inserting these into a database will not work. You will have to use the commandline parameter --replace, otherwise, if you dump your database and a row with id 6 in table table1 and try to import this into your other database, you'll get an error on duplicates if a row already has id = 6. Using the --replace parameter, it will overwrite older values, which can only happen if there is a new one (according to your WHERE clause).
So to quickly answer:
Yes, this will restore on the remote machine, but only if you saved using --replace (this will restore the latest version of the file you have)
I am not entirely sure if you can pipe backups. According to this website, you can, but I have never tried it before.