I try to restore a large table
pg_restore.exe -U postgres -d db_name --clean --if-exists --single-transaction F:\Backups\PostgreSQL\data.dump.gz
So I have a read lock for a few minutes. How to restore data with zero downtime for reading? I need only reading.
You would need to not do the --clean and instead do --data-only, but then do a DELETE from tablename inside the same transaction, before the COPY. I don't think there is a way to make pg_restore do this for you, but you could dump the output of pg_restore to a file and edit it, or use something like sed or perl to inject the DELETE.
This should work for table names which don't need to be quoted, and assuming none of the data being copied has first column which starts with 'COPY ':
pg_restore --data-only --single-transaction dmp.dmp -f -| perl -pe 's/^COPY ([\w.]+)/delete from $1; copy $1/' | psql -U postgres -d db_name
However, your schema changing method doesn't seem so dirty to me. It still requires a momentary access exclusive lock, so it isn't really zero downtime, but it might be unnoticable downtime if it can acquire said lock quickly enough.
Related
I have a dump file (size around 5 GB) which is taken via this command:
pg_dump -U postgres -p 5440 MYPRODDB > MYPRODDB_2022.dmp
The database consists multiple schemas (let's say Schema A,B,C and D) but i need to restore only one schema (schema A).
How can i achieve that? The command below didn't work and gave error:
pg_restore -U postgres -d MYPRODDB -n A -p 5440 < MYPRODDB_2022.dmp
pgrestore: error: input file appears to be a text format dump. please
use psql.
You cannot do that with a plain format dump. That's one of the reasons why you always use a different format unless you need an SQL script.
If you want to stick with a plain text dump:
pg_dump -U postgres -p 5440 -n A MYPRODDB > MYPRODDB_2022.dmp
psql -U postgres -d MYPRODDB -p 5440 -f MYPRODDB_2022.dmp
Though dumping back over the same database as above will throw errors unless you use --clean or its short form -c to create commands to drop existing objects before restoring them:
-c
--clean
Output commands to clean (drop) database objects prior to outputting the commands for creating them. (Unless --if-exists is also specified, restore might generate some harmless error messages, if any objects were not present in the destination database.)
This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.
Probably also a good idea to throw in --if-exists:
--if-exists
Use conditional commands (i.e., add an IF EXISTS clause) when cleaning database objects. This option is not valid unless --clean is also specified.
I want to restore a database from backup and rewrite all data that is there with backup data.
My current command is like this:
pg_restore -h localhost -U postgres -d dbName -v autobackup_file.dmp
How to restore and rewrite all data?
I've seen an option -c; is that the correct way?
And where should I put it in my command?
-c can be anywhere, e.g. immediately after pg_restore.
It will DROP all restored objects before restoring them, but it will not drop any objects that are not in the dump.
To drop and recreate the whole database so you get a clean copy, you can use -C -c.
After dumping a table and importing it to another postgres db constraints are missing.
I'm using this to dump:
pg_dump --host=local --username=user -W --encoding=UTF-8 -j 10 --file=dump_test --format=d -s --dbname=mydb -t addendum
This to import:
pg_restore -d myOtherdb --host=local -n public --username=user -W --exit-on-error --format=d -j 10 -t addendum dump_test/
What I can see in the resulting toc.dat is something like this:
ADD CONSTRAINT pk_addendum PRIMARY KEY (addendum_id);
> ALTER TABLE ONLY public.addendum DROP CONSTRAINT pk_addendum;
That looks like its creating and destroying the PK, but I'm not sure if my interpretation is correct as the file is binary.
edit: I'm using PostgreSQL 9.3
From the documentation:
Note: When -t is specified, pg_dump makes no attempt to dump any other database objects that the selected table(s) might depend upon. Therefore, there is no guarantee that the results of a specific-table dump can be successfully restored by themselves into a clean database.
You thus have some admittedly unattractive choices:
You can rebuild the constraints manually, especially if you still have the DDL which created them.
You can do a database-wide pg_dump to text, obtain the constraint DDL from there, see step 1.
You can do a database-wide pg_dump, and restore it fully.
I had the situation where the table already exists but using pg_restore deleted the constraints of the table.
There is an accepted answer already but I will try to provide an answer for those cases where the table to be restored is already available. In such cases, the constraints are deleted, only if you are trying to drop and recreate the table (-c or -C). Whereas if you only want the data from the dump you can perform delete all records on the table (DELETE FROM tableName) and then use pg_restore with -a flag. You can thus exclude -c or -C flag from you pg_restore command.
A little late to the party but here's something that may help.
If you're restoring a single table from a large dump file and having trouble getting the indexes with pg_restore (-t doesn't do indexes and constraints)
pg_restore db_dump_file.dump | awk '/table_name/{nr[NR]; nr[NR+1]}; NR in nr' > table_name_indexes_tmp.psql
You also need the subsequent line after a match for indexes and constraints. The awk command above gets line + 1 after every match.
This output file should contain your indexes (assuming the dump file actually contains them, plus data). Then you can apply them back to the table you restored as individual commands.
Not a perfect solution but better than trying to re-create them manually.
I have two databases on the same server. One named A and one named B. Booth databases have the same structure. I want to empty database B and load it with data from database A. Which is the best way to do this?
I have tried to take backup of database A in plain format. Then open the resulting sql-file and replace every occurence of 'A' with 'B' and then run the sql-script. This worked but I think it should be an easier way to move data from one database to another. Is it?
I use 'pgAdmin III' as my tool, but this is not necessary.
This is my first post here, hope the question is relevant and structured well enough. I tried google first but found it hard to find anyone with the same question.
Thanks in advance!
/David
SOLUTION: After help from Craig, this is how I did it
pg_dump -Fc -a -f a.dbbackup A
psql -c 'TRUNCATE table1, table2, ..., tableX CASCADE'
pg_restore dblive.backup -d B -c (not sure if -c was necessary)
Backup:
pg_dump -Fc -f a.dbbackup
Restore:
psql -c 'CREATE DATABASE b;'
pg_restore --dbname b a.dbbackup
Use the -U, -h etc options as required to connect to the correct host as the correct user with permissions to dump, create and restore the DB. See the docs for psql, pg_dump and pg_restore for more info (they all take the same options for connection control).
I have taken a dump of a database named temp1, by using the follwing command
$ pg_dump -i -h localhost -U postgres -F c -b -v -f pub.backup temp1
Now I want to restore the dump in a different database called "db_temp" , but in that I just want that all the tables should be created in a "temp_schema" ( not the default schema which is in the fms temp1 database ) which is in the "db_temp" database.
Is there any way to do this using pg_restore command?
Any other method also be appreciated!
A quick and dirty way:
1) rename default schema:
alter schema public rename to public_save;
2) create new schema as default schema:
create schema public;
3) restore data
pg_restore -f pub.backup db_temp [and whatever other options]
4) rename schemas according to need:
alter schema public rename to temp_schema;
alter schema public_save rename to public;
There is a simple solution:
Create your backup dump in plain SQL format (format "p" using the parameter --format=p or -F p)
Edit your pub.backup.sql dump with your favorite editor and add the following two lines at the top of your file:
create schema myschema;
SET search_path TO myschema;
Now you can restore your backup dump with the command
psql -f pub.backup.sql
The set search_path to <schema> command will set myschema as the default, so that new tables and other objects are created in this schema, independently of the "default" schema where they lived before.
There's no way in pg_restore itself. What you can do is use pg_restore to generate SQL output, and then send this through for example a sed script to change it. You need to be careful about how you write that sed script though, so it doesn't match and change things inside your data.
Probably the easiest method would be to simply rename the schema after restore, ie with the following SQL:
ALTER SCHEMA my_schema RENAME TO temp_schema
I believe that because you're using the compressed archive format for the output of pg_dump you can't alter it before restoring. The option would be to use the default output and do a search and replace on the schema name, but that would be risky and could perhaps cause data to be corrupted if you were not careful.
If you only have a few tables then you can restore one table at a time, pg_restore accepts -d database when you specify -t tablename. Of course, you'll have to set up the schema before restoring the tables and then sort out the indexes and constraints when you're done restoring the tables.
Alternatively, set up another server on a different port, restore using the new PostgreSQL server, rename the schema, dump it, and restore into your original database. This is a bit of a kludge of course but it will get the job done.
If you're adventurous you might be able to change the database name in the dump file using a hex editor. I think it is only mentioned in one place in the dump and as long as the new and old database names are the same it should work. YMMV, don't do anything like this in a production environment, don't blame me if this blows up and levels your home town, and all the rest of the usual disclaimers.
Rename the schema in a temporary database.
Export the schema:
pg_dump --schema-only --schema=prod > prod.sql
Create a new database. Restore the export:
psql -f prod.sql
ALTER SCHEMA prod RENAME TO somethingelse;
pg_dump --schema-only --schema=somethingelse > somethingelse.sql
(delete the database)
For the data you can just modify the set search_path at the top.
As noted, there's no direct support in pg_dump, psql or pg_restore to change the schema name during a dump/restore process. But it's fairly straightforward to export using "plain" format then modify the .sql file. This Bash script does the basics:
rename_schema () {
# Change search path so by default everything will go into the specified schema
perl -pi -e "s/SET search_path = $2, pg_catalog/SET search_path = $3, pg_catalog, $2;/" "$1"
# Change 'ALTER FUNCTION foo.' to 'ALTER FUNCTION bar.'
perl -pi -e 's/^([A-Z]+ [A-Z]+) '$2'\./$1 '$3'./' "$1"
# Change the final GRANT ALL ON SCHEMA foo TO PUBLIC
perl -pi -e 's/SCHEMA '$2'/SCHEMA '$3'/' "$1"
}
Usage:
pg_dump --format plain --schema=foo --file dump.sql MYDB
rename_schema dump.sql foo bar
psql -d MYDB -c 'CREATE SCHEMA bar;'
psql -d MYDB -f dumpsql
The question is pretty old, but maybe can help some one.
Streaming the output of pg_restore to sed and replace the schema name in order to import the dump to a different schema.
Something like:
pg_restore ${dumpfile} | \
sed -e "s/OWNER TO ${source_owner}/OWNER TO ${target_owner}/" \
-e "s/${source_schema}/${target_schema}/" | \
psql -h ${pgserver} -d ${dbname} -U ${pguser}