I have a table like this in the server:
CREATE TABLE example_table (
id BIGSERIAL PRIMARY KEY,
name VARCHAR(70) NOT NULL,
status VARCHAR(70) NOT NULL CONSTRAINT status_enum CHECK (status IN ('old', 'new')),
UNIQUE (id, name)
);
And I have an SQL file, example.sql. The first line contain a header:
name_of_class,status
'CLASSNAME','old';
And I try to run a psql \copy to google server:
PGPASSWORD=password psql -d database --username username --port 5432 --host 11.111.111 << EOF
BEGIN;
\copy example_table(name,status) FROM example.sql DELIMITER ',' CSV Header
COMMIT;
EOF
I then get this error:
ERROR: new row for relation "example_table" violates check constraint "status_enum"
DETAIL: Failing row contains (1, 'CLASSNAME', 'old';).
CONTEXT: COPY example_table, line 2: "'CLASSNAME','old';"
ROLLBACK
Any idea how to solve this? 🙂
It appears that your source csv is using the ' (single-quote) to quote all the columns. You could specify that as the quote character using the option QUOTE
The \copy command is trying to load 'old' into the status column that checks that values are either new or old. The extra quotes violate the constraint.
\copy example_table(name,status) FROM example.sql DELIMITER ',' CSV Header QUOTE ''''
4 single quotes are required because 1 specifies the actual quote char, 1 to escapes the quote-character, and 2 encloses the escaped quote-character.
Related
I am a beginner with PostgreSQL, I am trying to import a csv file into pgAdmin4, but seem to be having some trouble. I have the CSV file saved on my desktop and there is no header in the CSV file. Here is what my query currently looks like,
COPY opioid_csv(Substance, Source, Specific_Measure, Type_event, Region, PRUID, Time_Period, Year_Quarter, Aggregator, Disaggregator, Unit, Value)
FROM 'Users/myusername/Desktop/opioid_csv.csv'
DELIMITER ','
I have created a table with all the column names as well but I am getting the following error:
ERROR: relation "opioid_csv" does not exist SQL state: 42P01
Try, using psql:
postgres=# \dt *.opioid_csv
List of relations
Schema | Name | Type | Owner
--------------+------------+-------+-------
marcothesane | opioid_csv | table | marco
(1 row)
Very probably, if you did succeed to create your opioid_csv table, you created it in another schema than public. You would have to put the schema (marcothesane in my example) before the table name: use marcothesane.opioid_csv instead of just opioid.csv
Let me try to do it with a CSV file of mine:
marco ~/1/Vertica/supp $ head test.csv
id,text
1,1VIAgFg
2,IfPLHbT
3,EWOmXAx
4,zl8paoh
5,9EpQ9Kx
6,ZpagcCh
7,6xoVoit
8,mCniu1U
9,euieQZa
Matching table DDL - using my own d2l DDL inferrer:
marco ~/1/Vertica/supp $ d2lm -coldelcomma test.csv | tee test.ddl.sql
CREATE TABLE IF NOT EXISTS test (
id INTEGER NOT NULL
, text CHAR(7) NOT NULL
);
Then, I run psql with that newly generated script:
marco ~/1/Vertica/supp $ psql -af test.ddl.sql
CREATE TABLE IF NOT EXISTS test (
id INTEGER NOT NULL
, text CHAR(7) NOT NULL
);
CREATE TABLE
Then, as I have a header line in my file, I COPY with csv header:
marco ~/1/Vertica/supp $ psql -c "copy test from '/Users/marco/1/Vertica/supp/test.csv' delimiter ',' csv header"
COPY 50000
marco ~/1/Vertica/supp $ psql -c "select count(*) from test"
count
-------
50000
What is different from what you do?
System is Debian 11.2 with PostgreSQL 11.5.
I created a database and table as below:
CREATE DATABASE dbname OWNER=postgres
ENCODING= 'UTF8'
\c dbname
CREATE TABLE test(
id serial primary key,
site varchar(100) NOT NULL,
username char(30) NOT NULL,
password char(300) NOT NULL,
note varchar(200) DEFAULT NULL
);
Create bash file as below:
#!/bin/bash
res_user='me'
db_user='postgres'
db_name='dbname'
table_name='test'
sym_key='key'
#insert 4 columns
su $db_user <<EOFU
psql -d "$db_name" -U "$db_user" << EOF
INSERT INTO $table_name (site,username,password,note) VALUES ('v4','u3',pgp_sym_encrypt('password','key','cipher-algo=aes128,compress-algo=0,convert-crlf=1,sess-key=0,s2k-mode=3'),'note3');
EOF
EOFU
#column note has no output
password_arr=($(su $db_user <<EOFU
psql -tAq --field-separator= -d "$db_name" -U "$db_user" << EOF
SELECT "username",pgp_sym_decrypt(password::bytea,'key'),"note" FROM "$table_name" WHERE "site" LIKE '%v4%';
EOF
EOFU
))
echo "${password_arr[1]}" #output is passwordnote3
echo "${password_arr[2]}" #no ouput?
The expect output is:
${password_arr[1]} is `password`
${password_arr[2]} is `note3`
Run above bash script, but output "${password_arr[2]}" has no value,"${password_arr[1]}" is passwordnote3. Where is the problem?
I found the issue. The problem is you specified --field-separator to "nothing" instead of a space. It should be --field-separator=" ". This allowed the output of pgp_sym_decrypt() to concatenate with note. The username field however always had spaces probably since it has a fixed width of 30.
I also suggest that you reduce the number of row outputs to 1, and also enable "noglob" option when you're relying on word splitting. This can be done with set -f. You can also use read to get the needed fields. See How to split a string into an array in Bash?.
I'm trying to perform a db query through a docker inline command within a shell script.
myscript.sh:
docker run -it --rm -c "psql -U ${DB_USER} -d ${DB_NAME} -h ${DB_HOST}\
-c 'select col1, col2 , col3 from table1\
where table1.col2 = \"matching_text\" order by col1;'"
But I get an odd error:
ERROR: column "matching_text" does not exist
LINE 1: ...ndow where table1.col2 = "matching_t...
For some reason when I run this, psql thinks the matching_text in my query is referring to a column name. How would I get around this?
Note: Our database is implemented as a psql docker container.
The Postgres manual explains you need to use single quotes:
A string constant in SQL is an arbitrary sequence of characters bounded by single quotes ('), for example 'This is a string'. To include a single-quote character within a string constant, write two adjacent single quotes, e.g., 'Dianne''s horse'. Note that this is not the same as a double-quote character (").
See section 4.1.2.1 of the postgres manual.
Double quotes are for table or column identifiers:
There is a second kind of identifier: the delimited identifier or quoted identifier. It is formed by enclosing an arbitrary sequence of characters in double-quotes ("). A delimited identifier is always an identifier, never a key word. So "select" could be used to refer to a column or table named "select", whereas an unquoted select would be taken as a key word and would therefore provoke a parse error when used where a table or column name is expected. The example can be written with quoted identifiers like this:
UPDATE "my_table" SET "a" = 5;
See section 4.1.1 of the same manual.
Combination of post here and other post solved this issue:
Need to use single quotes for string query
Use double quotes for -c in psql command (Answer thread)
docker run -it --rm -c "psql -U ${DB_USER} -d ${DB_NAME} -h ${DB_HOST}\
-c \"select col1, col2 , col3 from table1\
where table1.col2 = 'matching_text' order by col1;\""
I want to convert my PostgreSQL table primary key UUID to character varying
ALTER TABLE payment_authorization ALTER COLUMN id TYPE VARCHAR;
When I run the above command showing below error, Beacause foreign key constraints failed. In my system have 200 tables. Is there any easy way to change all tables primary key?
Changing all the tables in place will probably be slow and cumbersome.
The easiest solution might be:
export the database with
pg_dump -F p -f dumpfile.sql dbname
replace uuid with text in the dump using an editor:
sed --in-place -e 's/uuid/text/g' dumpfile.sql
drop and re-create the database:
DROP DATABASE dbname;
CREATE DATABASE dbname;
import the dump:
psql -U postgres -d dbname -1 -f dumpfile.sql
I'm trying to copy a table from one database to another database (NOT schema). The code I used in terminal is as below:
pg_dump -U postgres -t OldSchema.TableToCopy OldDatabase | psql -U postgres -d NewDatabase
When I press Enter it requests postgres password I enter my pass and then It requests psql password. I enter it and press Enter. I receive lots of:
invalid command \N
ERROR: relation "TableToCopy" does not exist
Both tables have UTF8 encoding. Am I doing something wrong?
OS: windows XP
Error output:
psql:TblToCopy.sql:39236: invalid command \N
psql:TblToCopy.sql:39237: invalid command \N
psql:TblToCopy.sql:39238: invalid command \N
.
.
.
After Hundreds of above errors, the terminal echoes:
psql:TblToCopy.sql:39245: ERROR: syntax error at or near "509"
LINE 1: 509 some gibberish words and letters here
And Finally:
sql:TblToCopy.sql:39245: ERROR: relation "TableToCopy" does not exist
EDIT
I read this response to the same problem \N error with psql , it says to use INSERT instead of COPY, but in the file pg_dump created COPY. How to say to pg_dump to use INSERT instead of COPY?
I converted the file with iconv to utf-8. Now that error has gone but I have a new error. In this particular case when I use psql to import data to database something new happens. Table gets created but without data. It says:
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
psql:tblNew.sql:39610: ERROR: value too long for type character(3)
CONTEXT: COPY words, line 1, column first_two_letters: "سر"
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TRIGGER
I've tried to create a database with Encoding: UTF8 with a table and insert the two UTF-8 encoded characters the COPY command is trying to insert and it works when using INSERT.
CREATE DATABASE test
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
LC_COLLATE = 'English_United States.1252'
LC_CTYPE = 'English_United States.1252'
CONNECTION LIMIT = -1;
CREATE TABLE x
(
first_two_letters character(3)
)
WITH (
OIDS=FALSE
);
ALTER TABLE x
OWNER TO postgres;
INSERT INTO x(
first_two_letters)
VALUES ('سر');
According to http://rishida.net/tools/conversion/ for the failing COPY the Unicode code points are:
U+0633 U+0631
which are two characters, which means you should be able to store them in a column defined as character(3), which stores strings up to 3 characters (not bytes) in length.
and if we try to INSERT, it succeeds:
INSERT INTO x(
first_two_letters)
VALUES (U&'\0633\0631');
From the pgdump documentation you can INSERT instead of COPY by using the --inserts option
--inserts
Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can
be loaded into non-PostgreSQL databases. However, since this option
generates a separate command for each row, an error in reloading a row
causes only that row to be lost rather than the entire table contents.
Note that the restore might fail altogether if you have rearranged
column order. The --column-inserts option is safe against column order
changes, though even slower.
Try to use this instead for Step 1:
pg_dump -U postgres -t OldSchema."TableToCopy" --inserts OldDatabase > Table.sql
I've also tried to COPY from a table to a file and use COPY to import and for me it works.
Are you sure your client and server database encoding is UTF8 ?
Firstly, export the table named "x" from schema "public" on database "test" to a plain text SQL file:
pg_dump -U postgres -t public."x" test > x.sql
which creates the x.sql file that contains:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: x; Type: TABLE; Schema: public; Owner: postgres; Tablespace:
--
CREATE TABLE x (
first_two_letters character(3)
);
ALTER TABLE public.x OWNER TO postgres;
--
-- Data for Name: x; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY x (first_two_letters) FROM stdin;
سر
\.
--
-- PostgreSQL database dump complete
--
Secondly, import with:
psql -U postgres -d test -f x.sql
The table name should be quoted , as the following
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase | psql -U postgres -d NewDatabase
And I suggest you do the job in two steps
Step 1
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase > Table.sql
If step 1 goes ok then do the step2.
Step 2
psql -U postgres -d NewDatabase -f Table.sql