error during postgresql db backup restoration - postgresql

A database's backup file created in Windows 7 with:
pg_dump -U postgres -Fc [db_name] >D:\[db_backup_file].sql
Then I dropped it and restored it to test the process with:
pg_restore -U postgres -C -d postgres D:\[db_backup_file].sql
Everything worked fine.
However as I tried to restore it in Ubuntu 20.04 in a different device, I got an error:
could not execute query: ERROR: invalid locale name: (same as here)
So I followed the given instructions creating the database,
sudo -u postgres psql
create database [db_name];
and then I placed in the terminal the following command to restore backup:
pg_restore -U postgres -d postgres /home/../../[db_backup_file].sql
But again I got errors, as many were the tables, multiplied by four.
So for every table I get the following errors:
pg_restore: from TOC entry 315; 1259 29971 TABLE [table_name] postgres
pg_restore: error: could not execute query: ERROR: relation [table_name] already exists
Command was: CREATE TABLE public.[table_name] (
[pkey_column_name] integer NOT NULL,
.......
.......
.......
.......
.......
.......
);
pg_restore: from TOC entry 314; 1259 29969 SEQUENCE [table_name]_[pkey_column_name]_seq postgres
pg_restore: error: could not execute query: ERROR: relation "[table_name]_[pkey_column_name]_seq" already
exists
Command was: CREATE SEQUENCE public.[table_name]_[pkey_column_name]_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
pg_restore: from TOC entry 3522; 0 29971 TABLE DATA [table_name] postgres
pg_restore: error: COPY failed for table "[table_name]": ERROR: duplicate key value violates unique constraint "[table_name]_pkey"
DETAIL: Key ([pkey_column_name])=(1) already exists.
CONTEXT: COPY [table_name], line 1
pg_restore: from TOC entry 3267; 2606 29976 CONSTRAINT [table_name] [table_name]_pkey postgres
pg_restore: error:
could not execute query: ERROR: multiple primary keys for table "[table_name]" are not allowed
Command was: ALTER TABLE ONLY public.[table_name]
ADD CONSTRAINT [table_name]_pkey PRIMARY KEY ([pkey_column_name]);
When the tables were created the primary keys - if something has to do with it - were defined as auto increment, in the form of:
CREATE TABLE [table_name] (
[pkey_column_name] serial primary key,
.......
.......
.......
.......
.......
.......
);
Can anyone please help me about that?
EDIT: Actually the missing codepage type in the first error in my post yesterday was "Greek_Greece.1253". I used the locale -a command as you said, and I saw that among others my Ubuntu OS has en_US.UTF-8 and el_GR.UTF-8. So I'm wondering if the issue could be that incompatibility between Windows and Ubuntu character sets. If yes, how you thing that I could manage it?
Luckily the windows 7 device from which the backup files come from is still in use, and so the databases are active. But what I tried to create again the databases giving for LC_COLLATE and LC_CTYPE values compatible with ubuntu, didn't work.
EDIT 2: Finally it was that windows-linux incompatibility in characters encoding.
When I tried to use en_US.UTF-8 or el_GR.UTF-8 with encoding parameter as following:
pg_dump -E en_US.UTF-8 -U postgres -Fc [db_name] > D:\[backup_file].sql
I was getting:
pg_dump: invalid client encoding "en_US.UTF-8" specified
Then I tried to create in ubuntu the db before I restore it, under the command:
CREATE DATABASE database_name WITH ENCODING 'utf8' LC_COLLATE='el_GR.utf8' LC_CTYPE='el_GR.utf8' TEMPLATE template0;
and then:
pg_restore -U postgres -d postgres ~/../../backup_file.sql
but I got the same batch of errors I had in the initial post.
So the solution was to create a new database in windows, but now under 'C' char encoding (POSIX wouldn't be accepted), copy the tables from one database to another:
pg_dump -U postgres -t [table_name] [database_name] | psql -U postgres -d [database_name]
and then dump the newly created db, and restore it in ubuntu environment.

It could be that your Ubuntu OS does not have an en_US.UTF-8 locale. You can check for this by using this command in terminal:
locale -a # list all locales known to OS
If you cannot find the locale in the list, try making a new one according to this post
EDIT
With the additional information that the Windows encoding is Greek_Greece.1253, it still sounds like there is a mismatch. According to the pg_dump docs, you can explicitly set the encoding using the -E option. Probably you want to set it to something that Ubuntu can handle (i.e. en_US.UTF-8 or el_GR.UTF-8)
-E encoding
--encoding=encoding
Create the dump in the specified character set encoding. By default, the dump is
created in the database encoding. (Another way to get the same result is to set the
PGCLIENTENCODING environment variable to the desired dump encoding.)

Related

Restoring Postgres Database from backup does not work - public.uuid_generate_v1() does not exist

I am restoring a postgres database via .backup file from one postgres-14 instance to another:
pg_restore -h localhost -p 5432 -U postgres -d mydatabase -v "mybackupfile.backup"
The backup fails, complaining about
pg_restore: error: could not execute query: ERROR: function public.uuid_generate_v1() does not exist
However, the extension uuid-ossp that contains the respective function is installed on the target system.
What can I do about that?
The statment causing the error:
pg_restore: creating TABLE "data.mytable"
pg_restore: from TOC entry 215; 1259 155973 TABLE mytable superuser
pg_restore: error: could not execute query: ERROR: function public.uuid_generate_v1() does not exist
LINE 53: uuid uuid DEFAULT public.uuid_generate_v1(),
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
Command was: CREATE TABLE data.mytable (
my_id integer DEFAULT nextval('data.my_id_seq'::regclass) NOT NULL,
[...]
uuid uuid DEFAULT public.uuid_generate_v1()
);
I found the solution myself, but I am not sure which part of it was responsible:
In psql, connecting to mydbatabase and then calling the statement
SET search_path TO data, public;
calling the extension statement again:
create statement if not exists "uuid-ossp"
After that, the restore statement has been called successfully.
You need to add an extension;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
A similar answer here: LINK

pg_restore does not work with unknown error message

I tried to pg_dump from Postgresql 12.6 and pg_restore to Postgresql 14.5.
What I want is to move one table.
Here is the pg_dump command
pg_dump -Fc -b -U postgres -t test_eu3 -f C:\Users\...\Desktop\test_eu3.sql rulings
Here is the pg_restore command
pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
Here is a message after hitting pg_restore command.
C:\Users\...>pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
pg_restore: connecting to database for restore
Password:
pg_restore: implied data-only restore
pg_restore: processing data for table "public.test_eu3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2915; 0 3097390 TABLE DATA test_eu3 postgres
pg_restore: error: could not execute query: ERROR: ????????????"public.test_eu3"??????????s
Command was: COPY public.test_eu3 (id, "national", item_day, item_hs_all, item_hs2, item_hs4, item_hs6, item_place, image_amount, img_name_all, item_image, other_info, org_discription, org_discription2, eng_discription, eng_discription2, id2) FROM stdin;
pg_restore: warning: errors ignored on restore: 1
C:\Users\...>
It seems pg_dump woks without any problem, but pg_restore does not work with error message of "public.test_eu3" I don't know what it's indicating "public.test_eu3"?.
I've read pg_dump and pg_restore but it seems there is no solution for this problem.
How can I restore the database?
I just mentioned the above settings in this question but still if more code is required then tell me I'll update my question with that information. Thank you
Here is a message after hitting ALTER ROLE app_admin SET lc_messages = 'C'; command and restore.
postgres=# ALTER ROLE app_admin SET lc_messages = 'C';
ALTER ROLE
postgres=# \q
C:\Users\enosh>pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
pg_restore: connecting to database for restore
Password:
pg_restore: implied data-only restore
pg_restore: processing data for table "public.test_eu3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2915; 0 3097390 TABLE DATA test_eu3 postgres
pg_restore: error: could not execute query: ERROR: relation "public.test_eu3" does not exist
Command was: COPY public.test_eu3 (id, "national", item_day, item_hs_all, item_hs2, item_hs4, item_hs6, item_place, image_amount, img_name_all, item_image, other_info, org_discription, org_discription2, eng_discription, eng_discription2, id2) FROM stdin;
pg_restore: warning: errors ignored on restore: 1
The parameter lc_messages is set to a language that your terminal cannot display. There are two options:
If you connect to PostgreSQL with a superuser, you can use the PGOPTIONS environment variable to override the setting during your pg_restore. On Unix-like systems, that would work as follows:
export PGOPTIONS='-c lc_messages=C'
pg_restore ...
On Windows, you can use
set PGOPTIONS="-c lc_messages=C"
pg_restore ...
This will produce English error messages.
If you don't connect to PostgreSQL with a superuser, you are not allowed to change lc_messages. In that case, your only option is to ask the database administrator to change lc_messages in postgresql.conf (and reload PostgreSQL), or to set the parameter on your user:
ALTER ROLE app_admin SET lc_messages = 'C';

Table dump using pg_dump fails in AWS Redshift when table column includes reserved keyword

My goal is to dump the table structure for all tables in the database without any contents.
I have some Redshift tables that include reserved keywords. If I try to dump them I get this error:
pg_dump -sc --quote-all-identifiers -h example.com -p 5439 -d redacted -U redacted --table public.mytable > ./blah.txt
pg_dump: [archiver (db)] query failed: ERROR: syntax error at or near "stdout"
LINE 1: ... "status", "date") TO stdout;
^
pg_dump: [archiver (db)] query was: COPY "public"."mytable" ("status", "date") TO stdout;
If I try to export just the table structure using pg_dump -sc --quote-all-identifiers -h example.com -p 5439 -d redacted -U redacted --table public.mytable > ./blah.txt then the command succeeds, but the resulting file contains this create table statement:
CREATE TABLE "mytable" (
);
With nothing inside the parentheses aside from a new line.
The pg_dump version is pg_dump (PostgreSQL) 9.5.8 on Ubuntu 16.04.
The issue affects several different tables and one of the things in common among those tables is having columns named with words on the psql reserved words list. Maybe there's another explanation for why this is happening, so I'd love to hear alternate theories of the problem.
While it would be nice, I cannot change the names of the columns in the tables.
Strongly recommend that you shouldn't use pg_dump for Redshift. Try the UNLOAD command instead.
https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html

pg_restore certain tables only

I'm trying to write a pg_restore command to restore only certain tables (and their data) to my database.
Note: every command described begins with me dropping and re-creating the database and ends in: -v -x -O -j 8 -h localhost -U username -d database file.dump (For the curious, I didn't want to use --clean because the database that the dump came from has a different name.)
Since pg_restore works fine for me (with the above args), I looked at the pg_restore documentation, and tried something like this:
pg_restore -t table1 -t table2 ... (there are 121 tables I specify in this way).
However, I get errors like the following:
pg_restore: creating TABLE people
pg_restore: [archiver (db)] Error from TOC entry 123; 1234 12345 TABLE people dumped_table_username
pg_restore: [archiver (db)] could not execute query: ERROR: type "hstore" does not exist
LINE 14: extra_data hstore,
^
Command was: CREATE TABLE people (
id integer NOT NULL,
name string,
age integer,
date_of_birt...
I don't see why this would be an issue only when the -t flag is set, but it appears to be.
What's going on?
Edit: looks like this is a duplicate of pg_restore on table failing because of hstore, which was recently asked and has no accepted answer as of this time.
Apparently, pg_restore with the -t/--table flag set doesn't run CREATE EXTENSION commands that are in the dump file (because they're not technically part of that table). My problem was solved by manually running psql database -c "CREATE EXTENSION hstore;" before the pg_restore command.

Can't copy table to another database with pg_dump

I'm trying to copy a table from one database to another database (NOT schema). The code I used in terminal is as below:
pg_dump -U postgres -t OldSchema.TableToCopy OldDatabase | psql -U postgres -d NewDatabase
When I press Enter it requests postgres password I enter my pass and then It requests psql password. I enter it and press Enter. I receive lots of:
invalid command \N
ERROR: relation "TableToCopy" does not exist
Both tables have UTF8 encoding. Am I doing something wrong?
OS: windows XP
Error output:
psql:TblToCopy.sql:39236: invalid command \N
psql:TblToCopy.sql:39237: invalid command \N
psql:TblToCopy.sql:39238: invalid command \N
.
.
.
After Hundreds of above errors, the terminal echoes:
psql:TblToCopy.sql:39245: ERROR: syntax error at or near "509"
LINE 1: 509 some gibberish words and letters here
And Finally:
sql:TblToCopy.sql:39245: ERROR: relation "TableToCopy" does not exist
EDIT
I read this response to the same problem \N error with psql , it says to use INSERT instead of COPY, but in the file pg_dump created COPY. How to say to pg_dump to use INSERT instead of COPY?
I converted the file with iconv to utf-8. Now that error has gone but I have a new error. In this particular case when I use psql to import data to database something new happens. Table gets created but without data. It says:
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
psql:tblNew.sql:39610: ERROR: value too long for type character(3)
CONTEXT: COPY words, line 1, column first_two_letters: "سر"
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TRIGGER
I've tried to create a database with Encoding: UTF8 with a table and insert the two UTF-8 encoded characters the COPY command is trying to insert and it works when using INSERT.
CREATE DATABASE test
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
LC_COLLATE = 'English_United States.1252'
LC_CTYPE = 'English_United States.1252'
CONNECTION LIMIT = -1;
CREATE TABLE x
(
first_two_letters character(3)
)
WITH (
OIDS=FALSE
);
ALTER TABLE x
OWNER TO postgres;
INSERT INTO x(
first_two_letters)
VALUES ('سر');
According to http://rishida.net/tools/conversion/ for the failing COPY the Unicode code points are:
U+0633 U+0631
which are two characters, which means you should be able to store them in a column defined as character(3), which stores strings up to 3 characters (not bytes) in length.
and if we try to INSERT, it succeeds:
INSERT INTO x(
first_two_letters)
VALUES (U&'\0633\0631');
From the pgdump documentation you can INSERT instead of COPY by using the --inserts option
--inserts
Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can
be loaded into non-PostgreSQL databases. However, since this option
generates a separate command for each row, an error in reloading a row
causes only that row to be lost rather than the entire table contents.
Note that the restore might fail altogether if you have rearranged
column order. The --column-inserts option is safe against column order
changes, though even slower.
Try to use this instead for Step 1:
pg_dump -U postgres -t OldSchema."TableToCopy" --inserts OldDatabase > Table.sql
I've also tried to COPY from a table to a file and use COPY to import and for me it works.
Are you sure your client and server database encoding is UTF8 ?
Firstly, export the table named "x" from schema "public" on database "test" to a plain text SQL file:
pg_dump -U postgres -t public."x" test > x.sql
which creates the x.sql file that contains:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: x; Type: TABLE; Schema: public; Owner: postgres; Tablespace:
--
CREATE TABLE x (
first_two_letters character(3)
);
ALTER TABLE public.x OWNER TO postgres;
--
-- Data for Name: x; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY x (first_two_letters) FROM stdin;
سر
\.
--
-- PostgreSQL database dump complete
--
Secondly, import with:
psql -U postgres -d test -f x.sql
The table name should be quoted , as the following
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase | psql -U postgres -d NewDatabase
And I suggest you do the job in two steps
Step 1
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase > Table.sql
If step 1 goes ok then do the step2.
Step 2
psql -U postgres -d NewDatabase -f Table.sql