My goal is to dump the table structure for all tables in the database without any contents.
I have some Redshift tables that include reserved keywords. If I try to dump them I get this error:
pg_dump -sc --quote-all-identifiers -h example.com -p 5439 -d redacted -U redacted --table public.mytable > ./blah.txt
pg_dump: [archiver (db)] query failed: ERROR: syntax error at or near "stdout"
LINE 1: ... "status", "date") TO stdout;
^
pg_dump: [archiver (db)] query was: COPY "public"."mytable" ("status", "date") TO stdout;
If I try to export just the table structure using pg_dump -sc --quote-all-identifiers -h example.com -p 5439 -d redacted -U redacted --table public.mytable > ./blah.txt then the command succeeds, but the resulting file contains this create table statement:
CREATE TABLE "mytable" (
);
With nothing inside the parentheses aside from a new line.
The pg_dump version is pg_dump (PostgreSQL) 9.5.8 on Ubuntu 16.04.
The issue affects several different tables and one of the things in common among those tables is having columns named with words on the psql reserved words list. Maybe there's another explanation for why this is happening, so I'd love to hear alternate theories of the problem.
While it would be nice, I cannot change the names of the columns in the tables.
Strongly recommend that you shouldn't use pg_dump for Redshift. Try the UNLOAD command instead.
https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html
Related
I tried to pg_dump from Postgresql 12.6 and pg_restore to Postgresql 14.5.
What I want is to move one table.
Here is the pg_dump command
pg_dump -Fc -b -U postgres -t test_eu3 -f C:\Users\...\Desktop\test_eu3.sql rulings
Here is the pg_restore command
pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
Here is a message after hitting pg_restore command.
C:\Users\...>pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
pg_restore: connecting to database for restore
Password:
pg_restore: implied data-only restore
pg_restore: processing data for table "public.test_eu3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2915; 0 3097390 TABLE DATA test_eu3 postgres
pg_restore: error: could not execute query: ERROR: ????????????"public.test_eu3"??????????s
Command was: COPY public.test_eu3 (id, "national", item_day, item_hs_all, item_hs2, item_hs4, item_hs6, item_place, image_amount, img_name_all, item_image, other_info, org_discription, org_discription2, eng_discription, eng_discription2, id2) FROM stdin;
pg_restore: warning: errors ignored on restore: 1
C:\Users\...>
It seems pg_dump woks without any problem, but pg_restore does not work with error message of "public.test_eu3" I don't know what it's indicating "public.test_eu3"?.
I've read pg_dump and pg_restore but it seems there is no solution for this problem.
How can I restore the database?
I just mentioned the above settings in this question but still if more code is required then tell me I'll update my question with that information. Thank you
Here is a message after hitting ALTER ROLE app_admin SET lc_messages = 'C'; command and restore.
postgres=# ALTER ROLE app_admin SET lc_messages = 'C';
ALTER ROLE
postgres=# \q
C:\Users\enosh>pg_restore -v -U app_admin -d pre_rulings -t test_eu3 test_eu3.sql
pg_restore: connecting to database for restore
Password:
pg_restore: implied data-only restore
pg_restore: processing data for table "public.test_eu3"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2915; 0 3097390 TABLE DATA test_eu3 postgres
pg_restore: error: could not execute query: ERROR: relation "public.test_eu3" does not exist
Command was: COPY public.test_eu3 (id, "national", item_day, item_hs_all, item_hs2, item_hs4, item_hs6, item_place, image_amount, img_name_all, item_image, other_info, org_discription, org_discription2, eng_discription, eng_discription2, id2) FROM stdin;
pg_restore: warning: errors ignored on restore: 1
The parameter lc_messages is set to a language that your terminal cannot display. There are two options:
If you connect to PostgreSQL with a superuser, you can use the PGOPTIONS environment variable to override the setting during your pg_restore. On Unix-like systems, that would work as follows:
export PGOPTIONS='-c lc_messages=C'
pg_restore ...
On Windows, you can use
set PGOPTIONS="-c lc_messages=C"
pg_restore ...
This will produce English error messages.
If you don't connect to PostgreSQL with a superuser, you are not allowed to change lc_messages. In that case, your only option is to ask the database administrator to change lc_messages in postgresql.conf (and reload PostgreSQL), or to set the parameter on your user:
ALTER ROLE app_admin SET lc_messages = 'C';
A database's backup file created in Windows 7 with:
pg_dump -U postgres -Fc [db_name] >D:\[db_backup_file].sql
Then I dropped it and restored it to test the process with:
pg_restore -U postgres -C -d postgres D:\[db_backup_file].sql
Everything worked fine.
However as I tried to restore it in Ubuntu 20.04 in a different device, I got an error:
could not execute query: ERROR: invalid locale name: (same as here)
So I followed the given instructions creating the database,
sudo -u postgres psql
create database [db_name];
and then I placed in the terminal the following command to restore backup:
pg_restore -U postgres -d postgres /home/../../[db_backup_file].sql
But again I got errors, as many were the tables, multiplied by four.
So for every table I get the following errors:
pg_restore: from TOC entry 315; 1259 29971 TABLE [table_name] postgres
pg_restore: error: could not execute query: ERROR: relation [table_name] already exists
Command was: CREATE TABLE public.[table_name] (
[pkey_column_name] integer NOT NULL,
.......
.......
.......
.......
.......
.......
);
pg_restore: from TOC entry 314; 1259 29969 SEQUENCE [table_name]_[pkey_column_name]_seq postgres
pg_restore: error: could not execute query: ERROR: relation "[table_name]_[pkey_column_name]_seq" already
exists
Command was: CREATE SEQUENCE public.[table_name]_[pkey_column_name]_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
pg_restore: from TOC entry 3522; 0 29971 TABLE DATA [table_name] postgres
pg_restore: error: COPY failed for table "[table_name]": ERROR: duplicate key value violates unique constraint "[table_name]_pkey"
DETAIL: Key ([pkey_column_name])=(1) already exists.
CONTEXT: COPY [table_name], line 1
pg_restore: from TOC entry 3267; 2606 29976 CONSTRAINT [table_name] [table_name]_pkey postgres
pg_restore: error:
could not execute query: ERROR: multiple primary keys for table "[table_name]" are not allowed
Command was: ALTER TABLE ONLY public.[table_name]
ADD CONSTRAINT [table_name]_pkey PRIMARY KEY ([pkey_column_name]);
When the tables were created the primary keys - if something has to do with it - were defined as auto increment, in the form of:
CREATE TABLE [table_name] (
[pkey_column_name] serial primary key,
.......
.......
.......
.......
.......
.......
);
Can anyone please help me about that?
EDIT: Actually the missing codepage type in the first error in my post yesterday was "Greek_Greece.1253". I used the locale -a command as you said, and I saw that among others my Ubuntu OS has en_US.UTF-8 and el_GR.UTF-8. So I'm wondering if the issue could be that incompatibility between Windows and Ubuntu character sets. If yes, how you thing that I could manage it?
Luckily the windows 7 device from which the backup files come from is still in use, and so the databases are active. But what I tried to create again the databases giving for LC_COLLATE and LC_CTYPE values compatible with ubuntu, didn't work.
EDIT 2: Finally it was that windows-linux incompatibility in characters encoding.
When I tried to use en_US.UTF-8 or el_GR.UTF-8 with encoding parameter as following:
pg_dump -E en_US.UTF-8 -U postgres -Fc [db_name] > D:\[backup_file].sql
I was getting:
pg_dump: invalid client encoding "en_US.UTF-8" specified
Then I tried to create in ubuntu the db before I restore it, under the command:
CREATE DATABASE database_name WITH ENCODING 'utf8' LC_COLLATE='el_GR.utf8' LC_CTYPE='el_GR.utf8' TEMPLATE template0;
and then:
pg_restore -U postgres -d postgres ~/../../backup_file.sql
but I got the same batch of errors I had in the initial post.
So the solution was to create a new database in windows, but now under 'C' char encoding (POSIX wouldn't be accepted), copy the tables from one database to another:
pg_dump -U postgres -t [table_name] [database_name] | psql -U postgres -d [database_name]
and then dump the newly created db, and restore it in ubuntu environment.
It could be that your Ubuntu OS does not have an en_US.UTF-8 locale. You can check for this by using this command in terminal:
locale -a # list all locales known to OS
If you cannot find the locale in the list, try making a new one according to this post
EDIT
With the additional information that the Windows encoding is Greek_Greece.1253, it still sounds like there is a mismatch. According to the pg_dump docs, you can explicitly set the encoding using the -E option. Probably you want to set it to something that Ubuntu can handle (i.e. en_US.UTF-8 or el_GR.UTF-8)
-E encoding
--encoding=encoding
Create the dump in the specified character set encoding. By default, the dump is
created in the database encoding. (Another way to get the same result is to set the
PGCLIENTENCODING environment variable to the desired dump encoding.)
I have a script where pg_dump is failing with a message like this:
pg_dump -h db1 --format plain --encoding UTF8 --schema=public --schema-only --no-owner me
pg_dump: [archiver (db)] query failed: ERROR: permission denied for relation notmytable
pg_dump: [archiver (db)] query was: LOCK TABLE public.notmytable IN ACCESS SHARE MODE
This is causing the whole dump to abort.
Is there a way to either:
Ignore tables that aren't owned by our user?
Ignore errors?
I really don't want these tables in the dump, so even if we could get access to them, that wouldn't exactly solve the problem.
(Postgres 9.6.3)
It doesn't appear there is a standard way to do this, but using the --exclude-table flag, we can use a workaround:
export EXCLUDETABLE=$(psql -t -h $HOST -d $DBNAME -c "select '--exclude-table=' || string_agg(tablename,' --exclude-table=') FROM pg_catalog.pg_tables WHERE tableowner NOT LIKE 'myuser';" )
This sets EXCLUDETABLE to look like --exclude-table=foo --exclude-table=blah
Now we pass that to pg_dump:
echo Excluding these tables from dump: $EXCLUDETABLE
pg_dump -h $HOST --format plain --encoding UTF8 --schema=public --schema-only --no-owner $EXCLUDETABLE $DBNAME > public-schema.sql
I'm trying to write a pg_restore command to restore only certain tables (and their data) to my database.
Note: every command described begins with me dropping and re-creating the database and ends in: -v -x -O -j 8 -h localhost -U username -d database file.dump (For the curious, I didn't want to use --clean because the database that the dump came from has a different name.)
Since pg_restore works fine for me (with the above args), I looked at the pg_restore documentation, and tried something like this:
pg_restore -t table1 -t table2 ... (there are 121 tables I specify in this way).
However, I get errors like the following:
pg_restore: creating TABLE people
pg_restore: [archiver (db)] Error from TOC entry 123; 1234 12345 TABLE people dumped_table_username
pg_restore: [archiver (db)] could not execute query: ERROR: type "hstore" does not exist
LINE 14: extra_data hstore,
^
Command was: CREATE TABLE people (
id integer NOT NULL,
name string,
age integer,
date_of_birt...
I don't see why this would be an issue only when the -t flag is set, but it appears to be.
What's going on?
Edit: looks like this is a duplicate of pg_restore on table failing because of hstore, which was recently asked and has no accepted answer as of this time.
Apparently, pg_restore with the -t/--table flag set doesn't run CREATE EXTENSION commands that are in the dump file (because they're not technically part of that table). My problem was solved by manually running psql database -c "CREATE EXTENSION hstore;" before the pg_restore command.
How can I generate the DDL of a table programmatically on Postgresql? Is there a system query or command to do it? Googling the issue returned no pointers.
Use pg_dump with this options:
pg_dump -U user_name -h host database -s -t table_or_view_names -f table_or_view_names.sql
Description:
-s or --schema-only : Dump only ddl / the object definitions (schema), without data.
-t or --table Dump : Dump only tables (or views or sequences) matching table
Examples:
-- dump each ddl table elon build.
$ pg_dump -U elon -h localhost -s -t spacex -t tesla -t solarcity -t boring > companies.sql
Sorry if out of topic. Just wanna help who googling "psql dump ddl" and got this thread.
You can use the pg_dump command to dump the contents of the database (both schema and data). The --schema-only switch will dump only the DDL for your table(s).
Why would shelling out to psql not count as "programmatically?" It'll dump the entire schema very nicely.
Anyhow, you can get data types (and much more) from the information_schema (8.4 docs referenced here, but this is not a new feature):
=# select column_name, data_type from information_schema.columns
-# where table_name = 'config';
column_name | data_type
--------------------+-----------
id | integer
default_printer_id | integer
master_host_enable | boolean
(3 rows)
The answer is to check the source code for pg_dump and follow the switches it uses to generate the DDL. Somewhere inside the code there's a number of queries used to retrieve the metadata used to generate the DDL.
Here is a good article on how to get the meta information from information schema,
http://www.alberton.info/postgresql_meta_info.html.
I saved 4 functions to mock up pg_dump -s behaviour partially. Based on \d+ metacommand. The usage would be smth alike:
\pset format unaligned
select get_ddl_t(schemaname,tablename) as "--" from pg_tables where tableowner <> 'postgres';
Of course you have to create functions prior.
Working sample here at rextester