Can't copy table to another database with pg_dump - postgresql

I'm trying to copy a table from one database to another database (NOT schema). The code I used in terminal is as below:
pg_dump -U postgres -t OldSchema.TableToCopy OldDatabase | psql -U postgres -d NewDatabase
When I press Enter it requests postgres password I enter my pass and then It requests psql password. I enter it and press Enter. I receive lots of:
invalid command \N
ERROR: relation "TableToCopy" does not exist
Both tables have UTF8 encoding. Am I doing something wrong?
OS: windows XP
Error output:
psql:TblToCopy.sql:39236: invalid command \N
psql:TblToCopy.sql:39237: invalid command \N
psql:TblToCopy.sql:39238: invalid command \N
.
.
.
After Hundreds of above errors, the terminal echoes:
psql:TblToCopy.sql:39245: ERROR: syntax error at or near "509"
LINE 1: 509 some gibberish words and letters here
And Finally:
sql:TblToCopy.sql:39245: ERROR: relation "TableToCopy" does not exist
EDIT
I read this response to the same problem \N error with psql , it says to use INSERT instead of COPY, but in the file pg_dump created COPY. How to say to pg_dump to use INSERT instead of COPY?
I converted the file with iconv to utf-8. Now that error has gone but I have a new error. In this particular case when I use psql to import data to database something new happens. Table gets created but without data. It says:
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
psql:tblNew.sql:39610: ERROR: value too long for type character(3)
CONTEXT: COPY words, line 1, column first_two_letters: "سر"
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TRIGGER

I've tried to create a database with Encoding: UTF8 with a table and insert the two UTF-8 encoded characters the COPY command is trying to insert and it works when using INSERT.
CREATE DATABASE test
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
LC_COLLATE = 'English_United States.1252'
LC_CTYPE = 'English_United States.1252'
CONNECTION LIMIT = -1;
CREATE TABLE x
(
first_two_letters character(3)
)
WITH (
OIDS=FALSE
);
ALTER TABLE x
OWNER TO postgres;
INSERT INTO x(
first_two_letters)
VALUES ('سر');
According to http://rishida.net/tools/conversion/ for the failing COPY the Unicode code points are:
U+0633 U+0631
which are two characters, which means you should be able to store them in a column defined as character(3), which stores strings up to 3 characters (not bytes) in length.
and if we try to INSERT, it succeeds:
INSERT INTO x(
first_two_letters)
VALUES (U&'\0633\0631');
From the pgdump documentation you can INSERT instead of COPY by using the --inserts option
--inserts
Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can
be loaded into non-PostgreSQL databases. However, since this option
generates a separate command for each row, an error in reloading a row
causes only that row to be lost rather than the entire table contents.
Note that the restore might fail altogether if you have rearranged
column order. The --column-inserts option is safe against column order
changes, though even slower.
Try to use this instead for Step 1:
pg_dump -U postgres -t OldSchema."TableToCopy" --inserts OldDatabase > Table.sql
I've also tried to COPY from a table to a file and use COPY to import and for me it works.
Are you sure your client and server database encoding is UTF8 ?
Firstly, export the table named "x" from schema "public" on database "test" to a plain text SQL file:
pg_dump -U postgres -t public."x" test > x.sql
which creates the x.sql file that contains:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: x; Type: TABLE; Schema: public; Owner: postgres; Tablespace:
--
CREATE TABLE x (
first_two_letters character(3)
);
ALTER TABLE public.x OWNER TO postgres;
--
-- Data for Name: x; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY x (first_two_letters) FROM stdin;
سر
\.
--
-- PostgreSQL database dump complete
--
Secondly, import with:
psql -U postgres -d test -f x.sql

The table name should be quoted , as the following
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase | psql -U postgres -d NewDatabase
And I suggest you do the job in two steps
Step 1
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase > Table.sql
If step 1 goes ok then do the step2.
Step 2
psql -U postgres -d NewDatabase -f Table.sql

Related

Is it possible to make the psql \copy see a line inside a csv file as a comment?

I'm successfully inserting csv files to postgresql with the following command:
\COPY tablename(col1, col2, col3) FROM '/home/user/mycsv.txt' WITH CSV HEADER DELIMITER ';' NULL AS 'null';
However, I'd like to write some metadata inside this csv file with data that's repeatable. I know I could create a different file to store this data but I think it'd be a lot more convenient to store this metadata in the same csv file where the majority of the data is stored. I imagine a file like the following:
-- commented line with some metadata
col1;col2;col3
value;value;value
value;value;value
value;value;value
I've tried using -- /* /* and # as comments but the \copy command fails to import the data when I do that. Is there any way that I can tell the \copy psql command to see specific lines as comments just so I can insert data with lines that are not a part of the csv file? Is it possible?
Use the FROM PROGRAM construct to tell something else to filter them out.
\COPY tablename(col1, col2, col3) FROM PROGRAM 'egrep -v "^-- " mycsv.txt' WITH CSV HEADER DELIMITER ';' NULL AS 'null'
You see how pg_dump does it:
pg_dump -d test -U postgres -t orders -a -f test.sql
cat test.sql
--
-- PostgreSQL database dump
--
-- Dumped from database version 12.3
-- Dumped by pg_dump version 12.3
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Data for Name: orders; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY public.orders (order_id, total, order_date, user_id) FROM stdin;
1 100 2020-06-20 00:00:00 1
2 250 2020-06-20 00:00:00 2
\.
--
-- Name: orders_order_id_seq; Type: SEQUENCE SET; Schema: public; Owner: postgres
--
SELECT pg_catalog.setval('public.orders_order_id_seq', 2, true);
--
-- PostgreSQL database dump complete
--
psql -d test -U postgres -f test.sql
Null display is "NULL".
SET
SET
SET
SET
SET
set_config
------------
(1 row)
SET
SET
SET
SET
COPY 2
setval
--------
2
(1 row)

error during postgresql db backup restoration

A database's backup file created in Windows 7 with:
pg_dump -U postgres -Fc [db_name] >D:\[db_backup_file].sql
Then I dropped it and restored it to test the process with:
pg_restore -U postgres -C -d postgres D:\[db_backup_file].sql
Everything worked fine.
However as I tried to restore it in Ubuntu 20.04 in a different device, I got an error:
could not execute query: ERROR: invalid locale name: (same as here)
So I followed the given instructions creating the database,
sudo -u postgres psql
create database [db_name];
and then I placed in the terminal the following command to restore backup:
pg_restore -U postgres -d postgres /home/../../[db_backup_file].sql
But again I got errors, as many were the tables, multiplied by four.
So for every table I get the following errors:
pg_restore: from TOC entry 315; 1259 29971 TABLE [table_name] postgres
pg_restore: error: could not execute query: ERROR: relation [table_name] already exists
Command was: CREATE TABLE public.[table_name] (
[pkey_column_name] integer NOT NULL,
.......
.......
.......
.......
.......
.......
);
pg_restore: from TOC entry 314; 1259 29969 SEQUENCE [table_name]_[pkey_column_name]_seq postgres
pg_restore: error: could not execute query: ERROR: relation "[table_name]_[pkey_column_name]_seq" already
exists
Command was: CREATE SEQUENCE public.[table_name]_[pkey_column_name]_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
pg_restore: from TOC entry 3522; 0 29971 TABLE DATA [table_name] postgres
pg_restore: error: COPY failed for table "[table_name]": ERROR: duplicate key value violates unique constraint "[table_name]_pkey"
DETAIL: Key ([pkey_column_name])=(1) already exists.
CONTEXT: COPY [table_name], line 1
pg_restore: from TOC entry 3267; 2606 29976 CONSTRAINT [table_name] [table_name]_pkey postgres
pg_restore: error:
could not execute query: ERROR: multiple primary keys for table "[table_name]" are not allowed
Command was: ALTER TABLE ONLY public.[table_name]
ADD CONSTRAINT [table_name]_pkey PRIMARY KEY ([pkey_column_name]);
When the tables were created the primary keys - if something has to do with it - were defined as auto increment, in the form of:
CREATE TABLE [table_name] (
[pkey_column_name] serial primary key,
.......
.......
.......
.......
.......
.......
);
Can anyone please help me about that?
EDIT: Actually the missing codepage type in the first error in my post yesterday was "Greek_Greece.1253". I used the locale -a command as you said, and I saw that among others my Ubuntu OS has en_US.UTF-8 and el_GR.UTF-8. So I'm wondering if the issue could be that incompatibility between Windows and Ubuntu character sets. If yes, how you thing that I could manage it?
Luckily the windows 7 device from which the backup files come from is still in use, and so the databases are active. But what I tried to create again the databases giving for LC_COLLATE and LC_CTYPE values compatible with ubuntu, didn't work.
EDIT 2: Finally it was that windows-linux incompatibility in characters encoding.
When I tried to use en_US.UTF-8 or el_GR.UTF-8 with encoding parameter as following:
pg_dump -E en_US.UTF-8 -U postgres -Fc [db_name] > D:\[backup_file].sql
I was getting:
pg_dump: invalid client encoding "en_US.UTF-8" specified
Then I tried to create in ubuntu the db before I restore it, under the command:
CREATE DATABASE database_name WITH ENCODING 'utf8' LC_COLLATE='el_GR.utf8' LC_CTYPE='el_GR.utf8' TEMPLATE template0;
and then:
pg_restore -U postgres -d postgres ~/../../backup_file.sql
but I got the same batch of errors I had in the initial post.
So the solution was to create a new database in windows, but now under 'C' char encoding (POSIX wouldn't be accepted), copy the tables from one database to another:
pg_dump -U postgres -t [table_name] [database_name] | psql -U postgres -d [database_name]
and then dump the newly created db, and restore it in ubuntu environment.
It could be that your Ubuntu OS does not have an en_US.UTF-8 locale. You can check for this by using this command in terminal:
locale -a # list all locales known to OS
If you cannot find the locale in the list, try making a new one according to this post
EDIT
With the additional information that the Windows encoding is Greek_Greece.1253, it still sounds like there is a mismatch. According to the pg_dump docs, you can explicitly set the encoding using the -E option. Probably you want to set it to something that Ubuntu can handle (i.e. en_US.UTF-8 or el_GR.UTF-8)
-E encoding
--encoding=encoding
Create the dump in the specified character set encoding. By default, the dump is
created in the database encoding. (Another way to get the same result is to set the
PGCLIENTENCODING environment variable to the desired dump encoding.)

pg_dump and restore - pg_restore hangs on Windows

I have around 50gb of data in a postgres database on my laptop (mac) that I need to transfer to my new pc (windows). I've generated tar dumps using pg_dump of the schemas I need to transfer, but pg_restore just hangs.
To eliminate problems with the size of the file and the fact that the source is mac, I've boiled it down to the simplest test case I can find which is to create a new table in a new schema on my PC, export it using pg dump and then try to restore it back into the same database. Even with something this simple, pg_restore just hangs. I'm clearly missing something - probably quite obvious. Any ideas?
D:\Share\dbexport>psql -U postgres
Password for user postgres:
psql (12.1)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
postgres=# create schema new_schema
postgres-# create table new_schema.new_table(id numeric);
CREATE SCHEMA
postgres=# insert into new_schema.new_table values(1);
INSERT 0 1
postgres=# commit;
WARNING: there is no transaction in progress
COMMIT
postgres=# exit
Schema is created with new table and 1 row. So export
D:\Share\dbexport>pg_dump -U postgres -n new_schema -f new_schema_sql.sql
Password:
D:\Share\dbexport>more new_schema_sql.sql
--
-- PostgreSQL database dump
--
-- Dumped from database version 12.1
-- Dumped by pg_dump version 12.1
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Name: new_schema; Type: SCHEMA; Schema: -; Owner: postgres
--
CREATE SCHEMA new_schema;
ALTER SCHEMA new_schema OWNER TO postgres;
SET default_tablespace = '';
SET default_table_access_method = heap;
--
-- Name: new_table; Type: TABLE; Schema: new_schema; Owner: postgres
--
CREATE TABLE new_schema.new_table (
id numeric
);
ALTER TABLE new_schema.new_table OWNER TO postgres;
--
-- Data for Name: new_table; Type: TABLE DATA; Schema: new_schema; Owner: postgres
--
COPY new_schema.new_table (id) FROM stdin;
1
\.
So file has been created and has the expected content. I connect back to the database and drop the new schema before attempting the restore.
--
-- PostgreSQL database dump complete
--
D:\Share\dbexport>psql -U postgres
Password for user postgres:
psql (12.1)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
postgres=# drop schema new_schema cascade;
NOTICE: drop cascades to table new_schema.new_table
DROP SCHEMA
postgres=# select * from new_schema.new_table;
ERROR: relation "new_schema.new_table" does not exist
LINE 1: select * from new_schema.new_table;
^
postgres=# exit
D:\Share\dbexport>pg_restore -U postgres -f new_schema_sql.sql
And it just hangs at this last line. I'm a bit lost - I can't get pg_restore to output anything - I've tried with Verbose mode etc but nothing.
Does anyone know where I should be looking next?
David
So I will buy myself a dunce hat.
The issue as pointed out by #a_horse_with_no_name is that I misused the -f flag. That specifies the output file rather than the input file.
Using
pg_restore -U postgres -d postgres -n new_schema new_schema_custom.sql
fixed the issue. Thank you

Load sql file data in to single table in postgres

I have sql file(single_table_data.sql) that contains data of only one table(I have taken dump of only one table from server)
Now i have to insert this sql file in to only single table in my database,
so how to import sql file in to single table in postgres ?
Edit
For example i had database name SpeedData and table name CurrentTable, so now i want to
insert entire sql file data in to this table CurrentTable
Note: The sql file contains only insert statements(not even create statements)
From the documentation:
psql dbname < infile
This should create a new table with the name of the previously dumped one and insert all data into it. Replace dbname with the name of the new database and infile with the name/path of the file containing the dump, in your case (single_table_data.sql)
If the dump contains only the insert statements, create the table by hand and then execute
psql -U your_username -d dbname -f single_table_data.sql
You can simply import sql dump data into your postgres db.
if you have already created DB then don't need to follow 1st step:-
STEP=1
open terminal then run following commands to create postgres database and user:-
sudo -u postgres psql
postgres=# create database mydb;
postgres=# create user myuser with encrypted password 'mypass';
postgres=# grant all privileges on database mydb to myuser;
STEP=2
\c used for selecting your database.
postgres=# \c yourdatabasename
\i used for importing dump data in database.
yourdatabasename=# \i path_of_your_dump_file for example:-
yourdatabasename=# \i /home/developer/projects/django_projects/db_dump.sql
If you face this type of error when you importing data:-
ERROR: role "yourusername" does not exist
so you can make superuser to your "db_user/yourusername" using this command:-
postgres=# ALTER USER fusion WITH SUPERUSER;
ALTER ROLE
If I understood correctly you want to create table from file and fill in with data.
Correct command is
PGPASSWORD=<password> psql -f /home/.../filename.sql -h localhost -d database_name -U user_name

Dot (.) in Schema name makes pg_dump unusable

I have a schema named 2sample.sc. When I want to pg_dump some of its table, the following error appears:
pg_dump: No matching tables were found
My pg_dump command:
pg_dump -U postgres -t 2sample.sc."error_log" --inserts games > dump.sql
My pg_dump works fine on other schemas like 2sample.
What I did:
I tried to escape dot(.) with no success though
Use "schema.name.with.dots.in.it"."table.name.with.dots.in.it" to specify the schema.table:
-- test schema with a dot in its name
DROP SCHEMA "tmp.tmp" CASCADE;
CREATE SCHEMA "tmp.tmp" ;
SET search_path="tmp.tmp";
CREATE TABLE nononono
( dont SERIAL NOT NULL
);
insert into nononono
SELECT generate_series(1,10)
;
$pg_dump -t \"tmp.tmp\".\"nononono\" --schema-only -U postgres the_database
Output (snipped):
SET search_path = "tmp.tmp", pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: nononono; Type: TABLE; Schema: tmp.tmp; Owner: postgres; Tablespace:
--
CREATE TABLE nononono (
dont integer NOT NULL
);
BTW: why would you want to add a dot to a schema (or table) name? It is asking for trouble. The same for MixedCaseNames. Underscores work just fine.