How to insert string start with `'` to postgresql when `psql` in bash? - postgresql

#!/bin/bash
set -e
res_dir='/home/me'
db_port='5432'
db_name='test'
db_user='postgres'
db_password='passwoed'
table_name='record'
#input something start with \ will get error
read -p 'input site: ' site_input
res="'string";
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ($site_input,$res);
EOF
Above script will get error when your input string start with '.
input site: 'jh
ERROR: syntax error at or near "string"
LINE 1: INSERT INTO "record" (site,res) VALUES ('jh,'string);
^
I also tried other methods as below,but none workable.
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ('|| $site_input || ','|| $res || ');
EOF
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES (quote_literal($site_input),quote_literal($res));
EOF
How to insert string start with ' to postgresql when psql in bash?

You can escape single quotes by doubling them :
#!/bin/bash
set -e
res_dir='/home/me'
db_port='5432'
db_name='test'
db_user='postgres'
db_password='passwoed'
table_name='record'
#input something start with \ will get error
read -p 'input site: ' site_input
res="'string";
######## Doubling single quotes
site_input="${site_input//\'/\'\'}"
res="${res//\'/\'\'}"
######## Updated with '$site_input','$res'
psql postgresql://"$db_user":"$db_password"#localhost:"$db_port"/"$db_name" << EOF
INSERT INTO "$table_name" (site,res) VALUES ('$site_input','$res');
EOF
//\'/\'\' replaces all single quote ' by ''
For more information, see Parameter expansion

Related

Batch file for reading sql scripts from file and export results to csv

I want to make a batch file that will get query from .SQL script from the directory and export results in .csv format. I need to connect to the Postgres server.
So I'm trying to do this using that answer https://stackoverflow.com/a/39049102/9631920.
My file:
#!/bin/bash
# sql_to_csv.sh
echo test1
CONN="psql -U my_user -d my_db -h host -port"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' 'folder/folder/folder/file.sql' | tr '\n' ' ')"
echo test2
echo "$QUERY"
echo test3
echo "\\copy ($QUERY) to 'folder/folder/folder/file.csv' with csv header" | $CONN > /dev/null
echo query in progress
It shows me script from query and test3 and then stops. What am I doing wrong?
edit.
My file:
#!/bin/bash
PSQL = "psql -h 250.250.250.250 -p 5432 -U user -d test"
${PSQL} << OMG2
CREATE TEMP VIEW xyz AS
`cat C:\Users\test\Documents\my_query.sql`
;
\copy (select * from xyz) TO 'C:\Users\test\Documents\res.csv';
OMG2
But it's not asking password, and not getting any result file
a shell HERE-document will solve most of your quoting woes
a temp view will solve the single-query-on-a-single line problem
Example (using a multi-line two-table JOIN):
#!/bin/bash
PSQL="psql -U www twitters"
${PSQL} << OMG
-- Some comment here
CREATE TEMP VIEW xyz AS
SELECT twp.name, twt.*
FROM tweeps twp
JOIN tweets twt
ON twt.user_id = twp.id
AND twt.in_reply_to_id > 3
WHERE 1=1
AND (False OR twp.screen_name ilike '%omg%' )
;
\copy (select * from xyz) TO 'omg.csv';
OMG
If you want the contents of an existing .sql file, you can cat it inside the here document, using a backtick-expansion:
#!/bin/bash
PSQL="psql -X -n -U www twitters"
${PSQL} << OMG2
-- Some comment here
CREATE TEMP VIEW xyz AS
-- ... more comment
-- cat the original file here
`cat /home/dir1/dir2/dir3/myscript.sql`
;
\copy (select * from xyz) TO 'omg.csv';
OMG2

ERROR: invalid input syntax for type timestamp with time zone

I have a PostgreSQL query as below which is running fine . I am calling it from a shell script as below
Result=$(psql -U username -d database -t -c
$'SELECT round(sum(i.total), 2) AS "ROUND(sum(i.total), 2)"
FROM invoice i
WHERE i.create_datetime = '2019-03-01 00:00:00-06'
AND i.is_review = '1' AND i.user_id != 60;')
now I want the value which I have hard coded as i.create_datetime = '2019-03-01 00:00:00-06' to replace it with a variable date value.
I have tried two ways
way 1:
Result=$(psql -U username -d database -t -c
$'WITH var(reviewMonth) as (values(\'$reviewMonth\'))
SELECT round(sum(i.total),2) AS "ROUND(sum(i.total),2)"
FROM var,invoice i
WHERE i.create_datetime = var.reviewMonth::timestamp
AND i.is_review = \'1\' AND i.user_id != 60;')
and
way 2:
Result=$(psql -U username -d database -t -c
$'SELECT round(sum(i.total),2) AS "ROUND(sum(i.total),2)"
FROM invoice i
WHERE i.create_datetime = \'$reviewMonth\'
AND i.is_review = \'1\' AND i.user_id != 60;')
But both way it's throwing error
way 1 throwing error as :
ERROR: operator does not exist: timestamp with time zone = text
way 2 throwing error as :
ERROR: invalid input syntax for type timestamp with time zone: "$reviewMonth"
Please suggest what should be my approach.
You should try using the psql variables. Here's an example:
# Put the query in a file, with the variable TSTAMP:
> echo "SELECT :'TSTAMP'::timestamp with time zone;" > query.sql
> export TSTAMP='2019-03-01 00:00:00-06'
> RESULT=$(psql -U postgres -t --variable=TSTAMP="$TSTAMP" -f query.sql )
> echo $RESULT
2019-03-01 06:00:00+00
Note how we format the string literal substitution in the query: :'TSTAMP'
You could also do the substitution yourself. Here's an example using a heredoc:
> export TSTAMP='2019-03-01 00:00:01-06'
> RESULT=$(psql -U postgres -t << EOF
SELECT '$TSTAMP'::timestamp with time zone;
EOF
)
> echo $RESULT
2019-03-01 06:00:01+00
In this case, we aren't using psql's variable substitution, so we have to quote the variable like '$TSTAMP' . Using a heredoc makes the quoting much simpler than using -c because you aren't trying to quote the whole command.
EDIT: more examples because it appears this wasn't clear enough. TSTAMP does not have to be hard coded, it's just a bash variable than can be set like any other bash variable.
> TSTAMP=$(date -d 'now' +'%Y-%m-01 00:00:00')
> RESULT=$(psql -U postgres -t << EOF
SELECT '$TSTAMP'::timestamp with time zone;
EOF
)
> echo $RESULT
2019-06-01 00:00:00+00
However, if you're really just looking for the start of the month, there's no need for shell variables at all
> RESULT=$(psql -U postgres -t << EOF
SELECT date_trunc('month', now());
EOF
)
> echo $RESULT
2019-06-01 00:00:00+00

How to skip empty line in psql \COPY in PostgreSQL

In PostgreSQL psql, how to make \copy command ignore empty lines in input file?
Here is the code to reproduce it,
create table t1(
n1 int
);
echo "1
2
" > m.csv
psql> \copy t1(n1) FROM 'm.csv' (delimiter E'\t', NULL 'NULL', FORMAT CSV, HEADER false);
ERROR: invalid input syntax for integer: ""
CONTEXT: COPY t1, line 3, column n1: ""
There is an empty line in file m.csv
cat m.csv
1
2
<< empty line
PostgreSQL COPY is very strict, so there is not possibility to start COPY in tolerant mode. If it is possible, you can use COPY FROM PROGRAM
[pavel#nemesis ~]$ cat ~/data.csv
10,20,30
40,50,60
70,80,90
psql -c "\copy f from program ' sed ''/^\s*$/d'' ~/data.csv ' csv" postgres

Redis Mass Insertion from Postgresql file

Hello I am trying to migrate from Mysql to Postgresql.
I have an SQL file which queries some records and I want to put this in Redis with mass insert.
In Mysql it was working below this sample command;
sudo mysql -h $DB_HOST -u $DB_USERNAME -p$DB_PASSWORD $DB_DATABASE --skip-column-names --raw < test.sql | redis-cli --pipe
I figured out test.sql file for Postgresql syntax.
SELECT
'*3\r\n' ||
'$' || length(redis_cmd::text) || '\r\n' || redis_cmd::text || '\r\n' ||
'$' || length(redis_key::text) || '\r\n' || redis_key::text || '\r\n' ||
'$' || length(sval::text) || '\r\n' || sval::text || '\r\n'
FROM (
SELECT
'SET' as redis_cmd,
'ddi_lemmas:' || id::text AS redis_key,
lemma AS sval
FROM ddi_lemmas
) AS t
and its one output like
"*3\r\n$3\r\nSET\r\n$11\r\nddi_lemmas:1\r\n$22\r\nabil+abil+neg+aor+pagr\r\n"
But I couldn't find any example like Mysql command piping from command line.
There are some examples that have two stages not directly (first insert to a txt file and then put it in Redis)
sudo PGPASSWORD=$PASSWORD psql -U $USERNAME -h $HOSTNAME -d $DATABASE -f test.sql > data.txt
Above command working but with column names which i dont want.
I am trying to find directly send output of Postgresql result to Redis.
Could you help me please?
Solution:
If I want to insert with RESP commands from a sql file. (with the help of #teppic )
echo -e "$(psql -U $USERNAME -h $HOSTNAME -d $DATABASE -AEt -f test.sql)" | redis-cli --pipe
From the psql man page, -t will "Turn off printing of column names and result row count footers, etc."
-A turns off alignment, and -q sets "quiet" mode.
It looks like you're outputting RESP commands, in which case you'll have to use the escaped string format to get the newline/carriage return pairs, e.g. E'*3\r\n' (note the E).
It might be simpler to pipe SET commands to redis-cli:
psql -At -c "SELECT 'SET ddi_lemmas:' || id :: TEXT || ' ' || lemma FROM ddi_lemmas" | redis-cli

\echo is not working in heredoc pased to psql

Here is my sh file.
SCRIPT_DIR=`dirname $0`
export DATA_DIR=${SCRIPT_DIR}/data
export SQL_DIR=${SCRIPT_DIR}/sql
FILE_NAME=${DATA_DIR}/master_exec_alert_mails.dat
if [ ! -d $DATA_DIR ]
then
mkdir $DATA_DIR
fi
cd $SCRIPT_DIR
psql postgresql://xxxxx:xx#192.168.1.116:5432/xx -v ON_ERROR_STOP=1 << EOF > /dev/null
\o MASTER_EXECUTIVE_EFFORT_FILE_NAME
select to_char(LOCALTIMESTAMP-INTERVAL '8 DAY','Mon dd, yyyy') || ' - ' || to_char(LOCALTIMESTAMP-INTERVAL '2 DAY','Mon dd, yyyy')
\echo 'Master Exec effort list:'
\i master_exec_effort.sql
EOF
But it is not printing the message 'Master Exec effort list:' in the master_exec_alert_mails.dat output file.
Can anyone explain why it is not printing ?
Answer from Abelisto
Probably its because "\o [FILE] send all query results to file or |pipe". Instead of \echo try : "\qecho [STRING] write string to query output stream