# Dump my database to a tar file
pg_dump -f myDatabase.tar -F t -h myServer -U myUser -W -i myDatabase
# create a new database
createdb -h myServer -U myUser -T template0 myDatabaseCopy
# restore my database
pg_restore -d myDatabaseCopy -h myServer -U myUser myDatabase.tar
Then I get this error, and the import fails for an entire table.
psql:/home/me/myDatabase.tar:660266: ERROR: value too long for type character varying(100)
CONTEXT: COPY myTable, line 591, column myColumn: "A former member of the State Departmentâs âFuture of Iraqâ project and now at the Atlantic Cou..."
Those hat-a's are those annoying curly single and double quotes. It looks to me like they fit in the column at first, but somewhere in the export / import process they expand, and then no longer fit in the character varying(100) column.
I'm actually moving a live database on a server I have little permission for, so a sql only solution would be great. Is there a way to do something like
UPDATE myTable SET myColumn = removeNonAscii(myColumn) WHERE hasNonAscii(myColumn)
EDIT:
habe got it. I changed
createdb -h myServer -U myUser -T template0 myDatabaseCopy
to
createdb -h myServer -U myUser -T template0 -E UTF8 myDatabaseCopy
and that did the trick.
It seems that the problem is caused by database encoding.
For example, source database is encoded in “UTF-8” and destination is in more restricted character set like “SQL_ASCII.”
Check the encodings of both databases (\l of psql utility helps).
If they differ, recreate destination database with -Exxx option.
Related
I'm trying to copy a database from one remote server to another one. I've tried several different commands from my terminal (macOS):
"pg_dump -U postgres -d [DB] -f [DB].sql"
"pg_dump -U postgres -d [DB] -h [Host] -f [DB].sql"
But nothing works. I get errors like "pg_dump: error: connection to database [DB] failed: FATAL: database [DB] does not exist".
Any ideas how to solve this problem? I've tried to edit the pg_hba.conf, but it didn't work as well..
The procedure to make this work is :
First sub-option below outputs plain text file, second custom format(binary) file
a) pg_dump -C -h host_name -U user_name -d database_name -f database.sql
-C tells pg_dump to provide the command to create the database
on restore.
b) pg_dump -Fc -h host_name -U user_name -d database_name-f database.out
To restore you need different programs
a) For plain text option 1a do:
psql -d postgres -h host_name -U user_name -f database.sql
You need to connect to existing database, postgres in this case,
and then the commands from -C above will create
the database(database_name) and then connect to it for rest of operation.
b) For custom format option 1b:
pg_restore -C -d postgres -h host_name -U user_name database.out
Note: just specify the dump file(database.out) do not use -f.
More options and details can be found:
pg_dump
and
pg_restore
Look in the Notes section at the bottom of link for examples.
I have been using pg_dump for a while and every time I try to run the same script I seem to get issues. Not sure if it is user error or something to do with updating to Postgres 11.
Here is my command
pg_dump --dbname=postgresql://username:password#localhost:5432/DatabaseName --data-only --column-inserts -t "\"HoldingValuesTemp\"" > holdingValues.sql
This throws the error
pg_dump: too many command-line arguments (first is "HoldingValuesTemp\")
I think the issue has to do with the table name, it is case sensitive and is HoldingValuesTemp.
I tried to break it down into another pg_dump
pg_dump -d DatabaseName -p 5432 -U username --data-only --column-inserts -t "\"HoldingValuesTemp\"" > holdingValues.sql
Which gives the same error
So I also tried
pg_dump -d DatabaseName -p 5432 -U username --data-only --column-inserts -t '"HoldingValuesTemp"' > holdingValues.sql
after entering the password I get pg_dump: no matching tables were found
Any help would be appreciated.
My solution was more of a workaround than a solution.
The issue had to do with the table name, not sure why it was not finding that table but I assume it had to do with case sensitivity.
Solution:
Rename the table with a prefix that I looked up with a wildcard.
Table was "HoldingValuesTemp" I updated it to "ts_HoldingValuesTemp"
then ran the following command
pg_dump -d DatabaseName -p 5432 -U username --data-only --column-inserts -t 'ts_*' > holdingValues.sql
making it backup all tables that begin with "ts_"
Try to qualify the table name with the schema:
-t '"MySchema"."HoldingValuesTemp"'
There is also the possibility that you have a space character or similar in the table name.
I ONLY dump my databases as *.sql files not *.dump files. As a result NONE of the pg_restore commands work. I've been reading through answers and I swear most people have a reading disability lol
I am asking for the equivalent in psql for a common pg_restore commandLine method to restore a database.
I have no intention of dumping my databases as *.dump.
my question is this:
what is the equivalent to:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d my_db db/latest.dump
using psql
so...
something along the lines of:
psql --verbose --clean --no-acl --no-owner -h localhost -U myuser -d my_db db/latest.sql
With a SQL dump you need to decide whether you want to drop target objects, when dumping the database, not when importing it.
So, you need to use:
pg_dump --clean ....
Then the SQL dump will contain the necessary DROP statements.
Another option is to run drop owned by current_user before doing the import. This however requires that everything is owned by the user doing the import (so you can't run the import as e.g. postgres)
This can be combined with running the SQL dump:
psql -U your_user -d your_db -c 'drop owned by current_user' -f your_dump.sql
I use the postgres today
and got a problem
I dump the database that way
pg_dump zeus_development -U test > zeus_development.dump.out
what if I wnat to restore to another database zeus_production
How could I do?
Simple, first create your database using template0 as your template database:
createdb -U test -T template0 zeus_production
Then, restore your dump on this database:
psql -U test zeus_production -f /path/to/zeus_development.dump.out
When restoring, always use template0 explicit, as it is always an empty and unmodifiable database. If you don't use an explicit template, PostgreSQL will assume template1, and if it has some objects, like a table or function that your dumped database already has, you will get some errors while restoring.
Nonetheless, even if you were restoring on a database with the same name (zeus_development) you should create (or recreate) it the same way. Unless you used -C option while dumping (or -C of pg_restore if using a binary dump), which I don't recommend, because will give you less flexibility (like restoring on a different database name).
The PostgresSQL documentation has influenced me to use the custom format. I've been using it for years and it seems to have various advantages but your mileage may vary. That said, here is what worked for me:
pg_restore --no-owner --dbname postgres --create ~/Desktop/pg_dump
psql --dbname postgres -c 'ALTER DATABASE foodog_production RENAME TO foodog_development'
There was no foodog_development nor foodog_production databases existing before the sequence.
This restores the database from the dump (~/Desktop/pg_dump) which will create it with the name it was dumped as. The rename names the DB to whatever you want.
The --no-owner may not be needed if your user name is the same on both machines. In my case, the dump was done as user1 and the restore done as user2. The new objects need to be owned by user2 and --no-owner achieves this.
Isn't it easier to simply do the following?
createdb -U test -T zeus_development zeus_production
This has an answer on dba.stackexchange, which I reproduce here:
Let's define a few variables to make the rest easier to copy/paste
old_db=my_old_database
new_db=new_database_name
db_dump_file=backups/my_old_database.dump
user=postgres
The following assumes that your backup was created with the "custom" format like this:
pg_dump -U $user -F custom $old_db > "$db_dump_file"
To restore $db_dump_file to a new database name $new_db :
dropdb -U $user --if-exists $new_db
createdb -U $user -T template0 $new_db
pg_restore -U $user -d $new_db "$db_dump_file"
Here's a hacky way of doing it, that only works if you can afford the space and time to use regular .sql format, and if you can safely sed out your database name and user.
$ pg_dump -U my_production_user -h localhost my_production > my_prod_dump.sql
$ sed -i 's/my_production_user/my_staging_user/g' my_prod_dump.sql
$ sed -i 's/my_production/my_staging/g' my_prod_dump.sql
$ mv my_prod_dump.sql my_staging_dump.sql
$ sudo su postgres -c psql
psql> drop database my_staging;
psql> create database my_staging owner my_staging_user;
psql> \c my_staging;
psql> \i my_staging_dump.sql
If your dump does not include the name, the restore will use the DB defined in DESTINATION. Both SOURCE and DESTINATION are Connection URLs.
Dump without --create
pg_dump \
--clean --if-exists \
--file ${dump_path} \
--format=directory \
--jobs 5 \
--no-acl \
--no-owner \
${SOURCE}
Restore without --create
pg_restore \
--clean --if-exists \
--dbname=${DESTINATION} \
--format=directory \
--jobs=5 \
--no-acl \
--no-owner \
$dump_path
I am trying to setup a script to take a copy of a database from one server to another.
Thanks to this post Copying PostgreSQL database to another server I have found a way to do that.
But what I need to do is change the name of the database during the copy.
I have thought about using sed and doing a simple text replace. But I am worried that this could corrupt the database.
Does any one know the proper way of doing this?
As requested here are the commands I am using
pg_dump -C -U remoteuser -h remoteServer dbname | psql -h localhost -U localadmin template1
Just restore to a different database. For pg_restore of -Fc dumps from pg_dump's custom format:
createdb newdbname
pg_restore --dbname newdbname database.dump
For SQL-format dumps not created with the -C option to pg_dump:
createdb newdbname
psql -f database_dump.sql newdbname
If you're streaming the dump from a remote host, just omit -f database_dump.sql as the dump data is coming from stdin.
You can't easily CREATE DATABASE in the same command as your restore, because you need to connect to a different DB like template1 in order to create the new DB. So in your example you might:
psql -h localhost -U localadmin template1 -c 'CREATE DATABASE newdb;'
pg_dump -U remoteuser -h remoteServer dbname | psql -h localhost -U localadmin newdb
Note the omission of the -C flag to pg_dump.
The first command is just the longhand way of writing createdb -h localhost -U localadmin newdb.
Update: If you're stuck with a pg_dump created with the -C flag you can indeed just sed the dump so long as you're extremely careful. There should only be four lines (one a comment) at the start of the file that refer to the database name. For the database name "regress" dumped with Pg 9.1's pg_dump -C:
--
-- Name: regress; Type: DATABASE; Schema: -; Owner: craig
--
CREATE DATABASE regress WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
ALTER DATABASE regress OWNER TO craig;
\connect regress
This can be transformed quite safely with three (or four if you want to rewrite the comment) very specific sed commands. Do not just do a global find and replace on the database name, though.
sed \
-e 's/^CREATE DATABASE regress/CREATE DATABASE newdbname/' \
-e 's/^ALTER DATABASE regress/ALTER DATABASE newdbname/' \
-e 's/^\\connect regress/\\connect newdbname/' \
-e 's/^--Name: regress/--Name: newdbname/'
This should be a last resort; it's much better to just dump without -C.