How can I print text to stdout from a psql query? - postgresql

I have the following (massaged) psql script invoked from a bash shell:
foo=$( psql -q -t -R $'\x01' -F $'\x02' \
--variable="title=something" \
--variable="severity=level9" \
--pset='format=unaligned' \
<<'EOF'
SET standard_conforming_strings=on;
SET myvars.title = :title;
SET myvars.severity = :severity;
DO $$
BEGIN
IF EXISTS ( SELECT 1 from my_database
WHERE title=current_setting('myvars.title') \
AND severity=current_setting('myvars.severity') )
THEN
RAISE NOTICE 'Found existing entry';
ELSE
RAISE NOTICE 'Did not find existing entry';
END IF;
END;
$$;
EOF
)
I was hoping to capture in the bash variable "foo" some indication of whether or not the query succeeded. I thought I could add some kind of print/echo/return/whatever statement after each of the RAISE NOTICE statements to output 0 or 1 from the psql statement so it could be captured in foo for later evaluation in the bash script, e.g.:
foo=$( psql -q -t -R $'\x01' -F $'\x02' \
--variable="title=something" \
--variable="severity=level9" \
--pset='format=unaligned' \
<<'EOF'
SET standard_conforming_strings=on;
SET myvars.title = :title;
SET myvars.severity = :severity;
DO $$
BEGIN
IF EXISTS ( SELECT 1 from my_database
WHERE title=current_setting('myvars.title') \
AND severity=current_setting('myvars.severity') )
THEN
RAISE NOTICE 'Found existing entry';
magical_incantation 1;
ELSE
RAISE NOTICE 'Did not find existing entry';
magical_incantation 0;
END IF;
END;
$$;
EOF
)
I've been reading psql documentation for about the past 5 hours and cannot figure out the command/syntax of my imaginary magical_incantation above.
I am new to psql and tweaking someone else's code. I cannot change the main structure of the script, I just need some way to print a value.
Can anyone point me in the right direction?

\echo prints to stdout in psql. I think RAISE NOTICE goes to stderr.
It is possible to select into a psql variable and then echo it. A basic run-down is at How do you use script variables in PostgreSQL?
However I am not sure of any way to pull a query result into a variable at present.
Assuming that is not the answer you are looking for is just a SELECT.
So something like:
SELECT (EXISTS (....))::int;
-- true is 1, false is 0
Note there is no way to do this from a DO statement so you would have to actually create a function if you need to do this from plpgsql.

Related

Pipe commands from bash script to PSQL session [duplicate]

I want to execute a psql statement within a bash script and output the results to a file. The code I have below works as desired:
#!/bin/bash
query="select * from mytable;"
psql <<EOF > output.txt
\timing
$query
EOF
I want to run that psql command block 5 times and have the results appended to output.txt. It works fine if I just copy and paste it an additional 4 times, but when I try to put it inside of a for-loop, I get errors. Is there any way to do this?
This is the loop I tired:
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
psql <<EOF > output.txt
\timing
$query
EOF
done
If I move the final EOF all the way over to the left, it only executes once, as if the loop wasn't there.
You are overwriting the file each time with > inside the loop. You need >> inside or have > outside the loop:
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
psql <<EOF
\timing
$query
EOF
done > output.txt
Putting > after done is a little more efficient than >> inside the loop.
Similar post:
Pass values read from a file as input to an SQL query in Oracle
It is usually better to not run Postgres in a loop. Just generate the commands you want to execute, then run the sequence of generated commands once.
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
cat <<___EOF
\timing
$query
___EOF
done |
psql > output.txt
which of course in this case can be simplified to just
#!/bin/bash
printf '-- %s\n\\timing\nselect * from mytable;\n' {1..5} |
psql >output.txt
The brace expansion {1..5} is Bash-specific, so you can't use sh for this particular snippet. (There is a difference.)

Postgresql: Quoting variables passed from batch-file

I have a batch script that I want to be able to run on both my laptop and desktop. Depending on which machine I'm running it on, I need to pass 1 of 2 file paths to a postgresql script.
if "%computername%"=="LAPTOP" (
set init_path=C:\path1
) else (
set init_path=C:\path2
)
psql -U postgres -d dbname -v full_path=%init_path%\csvfile.csv -qf sql/sqlfile.sql
My postgresql script looks like this:
begin;
drop table if exists schm.table_name;
create table schm.table_name (
var1 date,
var2 float,
var3 text,
var4 float
);
truncate table schm.table_name;
\set quoted_path '\'' :full_path '\''
\copy schm.table_name from :quoted_path with delimiter as ',' csv quote as '"';
commit;
When I run it, I get the error:
:quoted_path: No such file or directory
Is this the correct way to quote a variable? I was basing my syntax on this thread.

psql - read SQL file and output to CSV

I have a SQL file my_query.sql:
select * from my_table
Using psql, I can read in this sql file:
\i my_query.sql
Or pass it in as an arg:
psql -f my_query.sql
And I can output the results of a query string to a csv:
\copy (select * from my_table) to 'output.csv' with csv header
Is there a way to combine these so I can output the results of a query from a SQL file to a CSV?
Unfortunately there's no baked-in functionality for this, so you need a little bash-fu to get this to work properly.
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' my_query.sql | tr '\n' ' ')"
echo "\\copy ($QUERY) to 'out.csv' with CSV HEADER" | $CONN
The sed fun removes all semicolons, comment lines, and end of line comments, and tr converts newlines to spaces (as mentioned in a comment by #abelisto):
-- my_query.sql
select *
from my_table
where timestamp < current_date -- only want today's records
limit 10;
becomes:
select * from my_table where timestamp < current_date limit 10
which then gets passed in to the valid psql command:
\copy (select * from my_table where timestamp < current_date) to 'out.csv' with csv header
Here's a script:
sql_to_csv.sh
#!/bin/bash
# sql_to_csv.sh
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' $1 | tr '\n' ' ')"
echo "$QUERY"
echo "\\copy ($QUERY) to '$2' with csv header" | $CONN > /dev/null
./sql_to_csv.sh my_query.sql out.csv
I think the simplest way is to take advantage of the shell's variable expansion capabilities:
psql -U my_user -d my_db -c "COPY ($(cat my_query.sql)) TO STDOUT WITH CSV HEADER" > my_query_results.csv
You could do it using a bash script.
dump_query_to_csv.sh:
#!/bin/bash
# Takes an sql query file as an argument and dumps its results
# to a CSV file using psql \copy command.
#
# Usage:
#
# dump_query_to_csv.sh <sql_query_file> [<csv_output_filesname>]
SQL_FILE=$1
[ -z $SQL_FILE ] && echo "Must supply query file" && exit
shift
OUT_FILE=$1
[ -z $OUT_FILE ] && OUT_FILE="output.csv" # default to "output.csv" if no argument is passed
TMP_TABLE=ttt_temp_table_xx # some table name that will not collide with existing tables
## Build a psql script to do the work
PSQL_SCRIPT=temp.psql
# create a temporary database table using the SQL from the query file
echo "DROP TABLE IF EXISTS $TMP_TABLE;CREATE TABLE $TMP_TABLE AS" > $PSQL_SCRIPT
cat $SQL_FILE >> $PSQL_SCRIPT
echo ";" >> $PSQL_SCRIPT
# copy the temporary table to the output CSV file
echo "\copy (select * from $TMP_TABLE) to '$OUT_FILE' with csv header" >> $PSQL_SCRIPT
# drop the temporary table
echo "DROP TABLE IF EXISTS $TMP_TABLE;" >> temp.sql
## Run psql script using psql
psql my_database < $PSQL_SCRIPT # replace my_database and add user login credentials as necessary
## Remove the psql script
rm $PSQL_SCRIPT
You'll need to edit the psql line in the script to connect to your database. The script could also be enhanced to take the database and account credentials as arguments.
The accepted solution is correct, but I had Windows and had to make it run via a batch (command) file. Posting it here if someone needs that
#echo off
echo 'Reading file %1'
set CONN="C:\Program Files\PostgreSQL\11\bin\psql.exe" -U dbusername -d mydbname
"C:\Program Files\Git\usr\bin\sed.exe" 's/;//g;/^--/ d;s/--.*//g;' %1 | "C:\Program Files\Git\usr\bin\tr.exe" '\n' ' ' > c:\temp\query.txt
set /p QUERY=<c:\temp\query.txt
echo %QUERY%
echo \copy (%QUERY%) to '%2' WITH (FORMAT CSV, HEADER) | %CONN%

How use GNU parallel and GNU SQL with \copy in PostgreSQL

I want to do a bulk load to a PostgreSQL database, there are several files and are pretty big. I just read in Using GNU Parallel With split about the GNU Parallel and GNU SQL, and It looks fantastic, Could some one help me with an example of using GNU Parallel, GNU SQL and \copy or COPY for doing a bulk load to PostgreSQL?
I'd just use pg_bulkload instead. It does that, and more, for you.
Both GNU parallel and pg_bulkload are cool, but not available in most default installations. I think this task can be achieved by using '&' (background sub-shell) and 'wait' in a shell script, which invokes multi \COPY operations simultaneously. Here is an example:
#!/bin/bash
PG_USER_NAME=bizusr
PG_DB_NAME=bizdb
BCP_DIR=/data/biz/bcp/input
do_bcp()
{
TABLE_NAME=$1
echo "`date` $$ copy $TABLE_NAME begin"
psql -q -U $PG_USER_NAME -d $PG_DB_NAME << EOF
-- SET DATESTYLE TO 'ISO,YMD'; -- you may need this when dealing with timestamps
\COPY $TABLE_NAME FROM '${BCP_DIR}/${TABLE_NAME}.bcp' WITH (FORMAT CSV, DELIMITER '|');
EOF
echo "`date` $$ copy $TABLE_NAME done"
}
echo "`date` $$ parallel copy started"
for BCP_FILE in `ls ${BCP_DIR}/*.bcp`; do
TABLE_NAME=`echo $BCP_FILE|awk -F"/" '{print $NF}'|sed -e s/\.bcp$//`
do_bcp $TABLE_NAME &
done
wait
echo "`date` $$ parallel copy finished"

PostgreSQL: How to pass parameters from command line?

I have a somewhat detailed query in a script that uses ? placeholders. I wanted to test this same query directly from the psql command line (outside the script). I want to avoid going in and replacing all the ? with actual values, instead I'd like to pass the arguments after the query.
Example:
SELECT *
FROM foobar
WHERE foo = ?
AND bar = ?
OR baz = ? ;
Looking for something like:
%> {select * from foobar where foo=? and bar=? or baz=? , 'foo','bar','baz' };
You can use the -v option e.g:
$ psql -v v1=12 -v v2="'Hello World'" -v v3="'2010-11-12'"
and then refer to the variables in SQL as :v1, :v2 etc:
select * from table_1 where id = :v1;
Please pay attention to how we pass string/date values using two quotes " '...' " But this way of interpolation is prone to SQL injections, because it's you who's responsible for quoting. E.g. need to include a single quote? -v v2="'don''t do this'".
A better/safer way is to let PostgreSQL handle it:
$ psql -c 'create table t (a int, b varchar, c date)'
$ echo "insert into t (a, b, c) values (:'v1', :'v2', :'v3')" \
| psql -v v1=1 -v v2="don't do this" -v v3=2022-01-01
Found out in PostgreSQL, you can PREPARE statements just like you can in a scripting language. Unfortunately, you still can't use ?, but you can use $n notation.
Using the above example:
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3 ;
EXECUTE foo('foo','bar','baz');
DEALLOCATE foo;
In psql there is a mechanism via the
\set name val
command, which is supposed to be tied to the -v name=val command-line option. Quoting is painful, In most cases it is easier to put the whole query meat inside a shell here-document.
Edit
oops, I should have said -v instead of -P (which is for formatting options) previous reply got it right.
You can also pass-in the parameters at the psql command-line, or from a batch file. The first statements gather necessary details for connecting to your database.
The final prompt asks for the constraint values, which will be used in the WHERE column IN() clause. Remember to single-quote if strings, and separate by comma:
#echo off
echo "Test for Passing Params to PGSQL"
SET server=localhost
SET /P server="Server [%server%]: "
SET database=amedatamodel
SET /P database="Database [%database%]: "
SET port=5432
SET /P port="Port [%port%]: "
SET username=postgres
SET /P username="Username [%username%]: "
SET /P bunos="Enter multiple constraint values for IN clause [%constraints%]: "
ECHO you typed %constraints%
PAUSE
REM pause
"C:\Program Files\PostgreSQL\9.0\bin\psql.exe" -h %server% -U %username% -d %database% -p %port% -e -v v1=%constraints% -f test.sql
Now in your SQL code file, add the v1 token within your WHERE clause, or anywhere else in the SQL. Note that the tokens can also be used in an open SQL statement, not just in a file. Save this as test.sql:
SELECT * FROM myTable
WHERE NOT someColumn IN (:v1);
In Windows, save the whole file as a DOS BATch file (.bat), save the test.sql in the same directory, and launch the batch file.
Thanks for Dave Page, of EnterpriseDB, for the original prompted script.
I would like to offer another answer inspired by #malcook's comment (using bash).
This option may work for you if you need to use shell variables within your query when using the -c flag. Specifically, I wanted to get the count of a table, whose name was a shell variable (which you can't pass directly when using -c).
Assume you have your shell variable
$TABLE_NAME='users'
Then you can get the results of that by using
psql -q -A -t -d databasename -c <<< echo "select count(*) from $TABLE_NAME;"
(the -q -A -t is just to print out the resulting number without additional formatting)
I will note that the echo in the here-string (the <<< operator) may not be necessary, I originally thought the quotes by themselves would be fine, maybe someone can clarify the reason for this.
It would appear that what you ask can't be done directly from the command line. You'll either have to use a user-defined function in plpgsql or call the query from a scripting language (and the latter approach makes it a bit easier to avoid SQL injection).
I've ended up using a better version of #vol7ron answer:
DO $$
BEGIN
IF NOT EXISTS(SELECT 1 FROM pg_prepared_statements WHERE name = 'foo') THEN
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3;
END IF;
END$$;
EXECUTE foo('foo','bar','baz');
This way you can always execute it in this order (the query prepared only if it does not prepared yet), repeat the execution and get the result from the last query.