normal insert:
insert into tfreeze(id,s) values(1,'foo');
I tried the following ways, both not working:
copy tfreeze(id,s ) from stdin;
1 foo
\.
copy tfreeze(id,s ) from stdin;
1 'foo'
\.
Only a few questions related from stdin in stackoverflow. https://stackoverflow.com/search?q=Postgres+Insert+statements+from+stdin
--
error code:
ERROR: 22P02: invalid input syntax for type integer: "1 foo"
CONTEXT: COPY tfreeze, line 1, column id: "1 foo"
LOCATION: pg_strtoint32, numutils.c:320
I get code from this(https://postgrespro.ru/education/books/internals) book.
code source: https://prnt.sc/eEsRZ5AK-tjQ
So far I tried:
1, foo, 1\t'foo', 1\tfoo
First, you have to use psql for that (you are already doing that).
You get that error because you use the default text format, which requires that the values are separated by tabulator characters (ASCII 9).
I recommend that you use the CSV format and separate the values with commas:
COPY tfreeze (id, s) FROM STDIN (FORMAT 'csv', FREEZE);
1,foo
\.
I want to make a batch file that will get query from .SQL script from the directory and export results in .csv format. I need to connect to the Postgres server.
So I'm trying to do this using that answer https://stackoverflow.com/a/39049102/9631920.
My file:
#!/bin/bash
# sql_to_csv.sh
echo test1
CONN="psql -U my_user -d my_db -h host -port"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' 'folder/folder/folder/file.sql' | tr '\n' ' ')"
echo test2
echo "$QUERY"
echo test3
echo "\\copy ($QUERY) to 'folder/folder/folder/file.csv' with csv header" | $CONN > /dev/null
echo query in progress
It shows me script from query and test3 and then stops. What am I doing wrong?
edit.
My file:
#!/bin/bash
PSQL = "psql -h 250.250.250.250 -p 5432 -U user -d test"
${PSQL} << OMG2
CREATE TEMP VIEW xyz AS
`cat C:\Users\test\Documents\my_query.sql`
;
\copy (select * from xyz) TO 'C:\Users\test\Documents\res.csv';
OMG2
But it's not asking password, and not getting any result file
a shell HERE-document will solve most of your quoting woes
a temp view will solve the single-query-on-a-single line problem
Example (using a multi-line two-table JOIN):
#!/bin/bash
PSQL="psql -U www twitters"
${PSQL} << OMG
-- Some comment here
CREATE TEMP VIEW xyz AS
SELECT twp.name, twt.*
FROM tweeps twp
JOIN tweets twt
ON twt.user_id = twp.id
AND twt.in_reply_to_id > 3
WHERE 1=1
AND (False OR twp.screen_name ilike '%omg%' )
;
\copy (select * from xyz) TO 'omg.csv';
OMG
If you want the contents of an existing .sql file, you can cat it inside the here document, using a backtick-expansion:
#!/bin/bash
PSQL="psql -X -n -U www twitters"
${PSQL} << OMG2
-- Some comment here
CREATE TEMP VIEW xyz AS
-- ... more comment
-- cat the original file here
`cat /home/dir1/dir2/dir3/myscript.sql`
;
\copy (select * from xyz) TO 'omg.csv';
OMG2
#!/bin/bash
set -e
res_dir='/home/me'
db_port='5432'
db_name='test'
db_user='postgres'
db_password='passwoed'
table_name='record'
#input something start with \ will get error
read -p 'input site: ' site_input
res="'string";
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ($site_input,$res);
EOF
Above script will get error when your input string start with '.
input site: 'jh
ERROR: syntax error at or near "string"
LINE 1: INSERT INTO "record" (site,res) VALUES ('jh,'string);
^
I also tried other methods as below,but none workable.
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ('|| $site_input || ','|| $res || ');
EOF
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES (quote_literal($site_input),quote_literal($res));
EOF
How to insert string start with ' to postgresql when psql in bash？
You can escape single quotes by doubling them :
#!/bin/bash
set -e
res_dir='/home/me'
db_port='5432'
db_name='test'
db_user='postgres'
db_password='passwoed'
table_name='record'
#input something start with \ will get error
read -p 'input site: ' site_input
res="'string";
######## Doubling single quotes
site_input="${site_input//\'/\'\'}"
res="${res//\'/\'\'}"
######## Updated with '$site_input','$res'
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ('$site_input','$res');
EOF
//\'/\'\' replaces all single quote ' by ''
For more information, see Parameter expansion
I have a requirement to dump the contents of a definable selection of tables as CSV's for an initial load of systems that are not able to connect with PostgreSQL for various reasons.
I have written a script to do this which runs through a list of tables using psql with the -c flag to run psql's \COPY command to dump the corresponding table to a file like this:
COPY table_name TO table_name.csv WITH (FORMAT 'csv', HEADER, QUOTE '\"', DELIMITER '|');
It works fine. But I am sure you have already spotted the problem: as the process takes ~57 minutes for ~60 odd tables, the likelyhood of consistency is quite close to absolute zero.
I had a think about it and suspected I could make a few lightweight changes to pg_dump to do what I want, i.e., create multiple csv's from pg_dump whilst having a hope of integrity between the tables - and being able to specify parallel dumps too.
I have added a few flags to allow me to apply a file postfix (the date), set the format options and pass in a path for the relevant output file.
However my modified pg_dump was failing when writing to a file, like:
COPY table_name (pkey_id, field1, field2 ... fieldn) TO table_name.csv WITH (FORMAT 'csv', HEADER, QUOTE '"', DELIMITER '|')
Note: Within pg_dump, the column list is expanded
So I cast around for further information and found these COPY Tips.
It looks like writing to a file is a no-no over the network; however I am on the same machine (for now). I felt writing to /tmp would be OK as it is writable by anyone.
So I tried cheating with:
seingramp#seluonkeydb01:~$ ./tp_dump -a -t table_name -D /tmp/ -k "FORMAT 'csv', HEADER, QUOTE '\"', DELIMITER '|'" -K "_$DATE_POSTFIX"
tp_dump: warning: there are circular foreign-key constraints on this table:
tp_dump: table_name
tp_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
tp_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.
--
-- PostgreSQL database dump
--
-- Dumped from database version 12.3
-- Dumped by pg_dump version 14devel
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Data for Name: material_master; Type: TABLE DATA; Schema: mm; Owner: postgres
--
COPY table_name (pkey_id, field1, field2 ... fieldn) FROM stdin;
tp_dump: error: query failed:
tp_dump: error: query was: COPY table_name (pkey_id, field1, field2 ... fieldn) TO PROGRAM 'gzip > /tmp/table_name_20200814.csv.gz' WITH (FORMAT 'csv', HEADER, QUOTE '"', DELIMITER '|')
I have neutered the data as it is customer specific.
I didn't find pg_dump's error message very helpful, do you have any ideas as to what I am doing wrong?
The changes really are quite small (excuse the code!) starting ~line 1900, ignoring the flags added around getopt().
/*
* Use COPY (SELECT ...) TO when dumping a foreign table's data, and when
* a filter condition was specified. For other cases a simple COPY
* suffices.
*/
if (tdinfo->filtercond || tbinfo->relkind == RELKIND_FOREIGN_TABLE)
{
/* Note: this syntax is only supported in 8.2 and up */
appendPQExpBufferStr(q, "COPY (SELECT ");
/* klugery to get rid of parens in column list */
if (strlen(column_list) > 2)
{
appendPQExpBufferStr(q, column_list + 1);
q->data[q->len - 1] = ' ';
}
else
appendPQExpBufferStr(q, "* ");
if ( copy_from_spec )
{
if ( copy_from_postfix )
{
appendPQExpBuffer(q, "FROM %s %s) TO PROGRAM 'gzip > %s%s%s.csv.gz' WITH (%s)",
fmtQualifiedDumpable(tbinfo),
tdinfo->filtercond ? tdinfo->filtercond : "",
copy_from_dest ? copy_from_dest : "",
fmtQualifiedDumpable(tbinfo),
copy_from_postfix,
copy_from_spec);
}
else
{
appendPQExpBuffer(q, "FROM %s %s) TO PROGRAM 'gzip > %s%s.csv.gz' WITH (%s)",
fmtQualifiedDumpable(tbinfo),
tdinfo->filtercond ? tdinfo->filtercond : "",
copy_from_dest ? copy_from_dest : "",
fmtQualifiedDumpable(tbinfo),
copy_from_spec);
}
}
else
{
appendPQExpBuffer(q, "FROM %s %s) TO stdout;",
fmtQualifiedDumpable(tbinfo),
tdinfo->filtercond ? tdinfo->filtercond : "");
}
}
else
{
if ( copy_from_spec )
{
if ( copy_from_postfix )
{
appendPQExpBuffer(q, "COPY %s %s TO PROGRAM 'gzip > %s%s%s.csv.gz' WITH (%s)",
fmtQualifiedDumpable(tbinfo),
column_list,
copy_from_dest ? copy_from_dest : "",
fmtQualifiedDumpable(tbinfo),
copy_from_postfix,
copy_from_spec);
}
else
{
appendPQExpBuffer(q, "COPY %s %s TO PROGRAM 'gzip > %s%s.csv.gz' WITH (%s)",
fmtQualifiedDumpable(tbinfo),
column_list,
copy_from_dest ? copy_from_dest : "",
fmtQualifiedDumpable(tbinfo),
copy_from_spec);
}
}
else
{
appendPQExpBuffer(q, "COPY %s %s TO stdout;",
fmtQualifiedDumpable(tbinfo),
column_list);
}
I tried a couple of other cheats too, like specifying a directory owned by postgres. I know it's a quick hack but I hope you can help, and thanks for looking.
This is a use case for pg_restore -f.
So:
-- Create custom format dump file
pg_dump -d some_db -U some_user -Fc -f dump.out
-- Move that file to where you need it
-- Dump data only from named table to a file from the dump file.
pg_restore -a -t table_1 -f table_1_data.sql dump.out
The pg_dump will create a consistent snapshot of the tables, so you have the database in a 'frozen' state in dump.out. Then you can use pg_restore to 'thaw out' those parts you need on your schedule. By using -a you will get the COPY you want.
Hello I am trying to migrate from Mysql to Postgresql.
I have an SQL file which queries some records and I want to put this in Redis with mass insert.
In Mysql it was working below this sample command;
sudo mysql -h $DB_HOST -u $DB_USERNAME -p$DB_PASSWORD $DB_DATABASE --skip-column-names --raw < test.sql | redis-cli --pipe
I figured out test.sql file for Postgresql syntax.
SELECT
'*3\r\n' ||
'$' || length(redis_cmd::text) || '\r\n' || redis_cmd::text || '\r\n' ||
'$' || length(redis_key::text) || '\r\n' || redis_key::text || '\r\n' ||
'$' || length(sval::text) || '\r\n' || sval::text || '\r\n'
FROM (
SELECT
'SET' as redis_cmd,
'ddi_lemmas:' || id::text AS redis_key,
lemma AS sval
FROM ddi_lemmas
) AS t
and its one output like
"*3\r\n$3\r\nSET\r\n$11\r\nddi_lemmas:1\r\n$22\r\nabil+abil+neg+aor+pagr\r\n"
But I couldn't find any example like Mysql command piping from command line.
There are some examples that have two stages not directly (first insert to a txt file and then put it in Redis)
sudo PGPASSWORD=$PASSWORD psql -U $USERNAME -h $HOSTNAME -d $DATABASE -f test.sql > data.txt
Above command working but with column names which i dont want.
I am trying to find directly send output of Postgresql result to Redis.
Could you help me please?
Solution:
If I want to insert with RESP commands from a sql file. (with the help of #teppic )
echo -e "$(psql -U $USERNAME -h $HOSTNAME -d $DATABASE -AEt -f test.sql)" | redis-cli --pipe
From the psql man page, -t will "Turn off printing of column names and result row count footers, etc."
-A turns off alignment, and -q sets "quiet" mode.
It looks like you're outputting RESP commands, in which case you'll have to use the escaped string format to get the newline/carriage return pairs, e.g. E'*3\r\n' (note the E).
It might be simpler to pipe SET commands to redis-cli:
psql -At -c "SELECT 'SET ddi_lemmas:' || id :: TEXT || ' ' || lemma FROM ddi_lemmas" | redis-cli