I want to execute a psql statement within a bash script and output the results to a file. The code I have below works as desired:
#!/bin/bash
query="select * from mytable;"
psql <<EOF > output.txt
\timing
$query
EOF
I want to run that psql command block 5 times and have the results appended to output.txt. It works fine if I just copy and paste it an additional 4 times, but when I try to put it inside of a for-loop, I get errors. Is there any way to do this?
This is the loop I tired:
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
psql <<EOF > output.txt
\timing
$query
EOF
done
If I move the final EOF all the way over to the left, it only executes once, as if the loop wasn't there.
You are overwriting the file each time with > inside the loop. You need >> inside or have > outside the loop:
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
psql <<EOF
\timing
$query
EOF
done > output.txt
Putting > after done is a little more efficient than >> inside the loop.
Similar post:
Pass values read from a file as input to an SQL query in Oracle
It is usually better to not run Postgres in a loop. Just generate the commands you want to execute, then run the sequence of generated commands once.
#!/bin/bash
query="select * from mytable;"
for (( i=0; i<5; i++ ))
do
cat <<___EOF
\timing
$query
___EOF
done |
psql > output.txt
which of course in this case can be simplified to just
#!/bin/bash
printf '-- %s\n\\timing\nselect * from mytable;\n' {1..5} |
psql >output.txt
The brace expansion {1..5} is Bash-specific, so you can't use sh for this particular snippet. (There is a difference.)
Related
the following code is correct ?
i want to store foo table output to file.txt file how to write shell script for this
psql <<EOF
\pset format wrapped
SELECT * FROM foo;
EOF |tee -a file.txt
I have a SQL file my_query.sql:
select * from my_table
Using psql, I can read in this sql file:
\i my_query.sql
Or pass it in as an arg:
psql -f my_query.sql
And I can output the results of a query string to a csv:
\copy (select * from my_table) to 'output.csv' with csv header
Is there a way to combine these so I can output the results of a query from a SQL file to a CSV?
Unfortunately there's no baked-in functionality for this, so you need a little bash-fu to get this to work properly.
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' my_query.sql | tr '\n' ' ')"
echo "\\copy ($QUERY) to 'out.csv' with CSV HEADER" | $CONN
The sed fun removes all semicolons, comment lines, and end of line comments, and tr converts newlines to spaces (as mentioned in a comment by #abelisto):
-- my_query.sql
select *
from my_table
where timestamp < current_date -- only want today's records
limit 10;
becomes:
select * from my_table where timestamp < current_date limit 10
which then gets passed in to the valid psql command:
\copy (select * from my_table where timestamp < current_date) to 'out.csv' with csv header
Here's a script:
sql_to_csv.sh
#!/bin/bash
# sql_to_csv.sh
CONN="psql -U my_user -d my_db"
QUERY="$(sed 's/;//g;/^--/ d;s/--.*//g;' $1 | tr '\n' ' ')"
echo "$QUERY"
echo "\\copy ($QUERY) to '$2' with csv header" | $CONN > /dev/null
./sql_to_csv.sh my_query.sql out.csv
I think the simplest way is to take advantage of the shell's variable expansion capabilities:
psql -U my_user -d my_db -c "COPY ($(cat my_query.sql)) TO STDOUT WITH CSV HEADER" > my_query_results.csv
You could do it using a bash script.
dump_query_to_csv.sh:
#!/bin/bash
# Takes an sql query file as an argument and dumps its results
# to a CSV file using psql \copy command.
#
# Usage:
#
# dump_query_to_csv.sh <sql_query_file> [<csv_output_filesname>]
SQL_FILE=$1
[ -z $SQL_FILE ] && echo "Must supply query file" && exit
shift
OUT_FILE=$1
[ -z $OUT_FILE ] && OUT_FILE="output.csv" # default to "output.csv" if no argument is passed
TMP_TABLE=ttt_temp_table_xx # some table name that will not collide with existing tables
## Build a psql script to do the work
PSQL_SCRIPT=temp.psql
# create a temporary database table using the SQL from the query file
echo "DROP TABLE IF EXISTS $TMP_TABLE;CREATE TABLE $TMP_TABLE AS" > $PSQL_SCRIPT
cat $SQL_FILE >> $PSQL_SCRIPT
echo ";" >> $PSQL_SCRIPT
# copy the temporary table to the output CSV file
echo "\copy (select * from $TMP_TABLE) to '$OUT_FILE' with csv header" >> $PSQL_SCRIPT
# drop the temporary table
echo "DROP TABLE IF EXISTS $TMP_TABLE;" >> temp.sql
## Run psql script using psql
psql my_database < $PSQL_SCRIPT # replace my_database and add user login credentials as necessary
## Remove the psql script
rm $PSQL_SCRIPT
You'll need to edit the psql line in the script to connect to your database. The script could also be enhanced to take the database and account credentials as arguments.
The accepted solution is correct, but I had Windows and had to make it run via a batch (command) file. Posting it here if someone needs that
#echo off
echo 'Reading file %1'
set CONN="C:\Program Files\PostgreSQL\11\bin\psql.exe" -U dbusername -d mydbname
"C:\Program Files\Git\usr\bin\sed.exe" 's/;//g;/^--/ d;s/--.*//g;' %1 | "C:\Program Files\Git\usr\bin\tr.exe" '\n' ' ' > c:\temp\query.txt
set /p QUERY=<c:\temp\query.txt
echo %QUERY%
echo \copy (%QUERY%) to '%2' WITH (FORMAT CSV, HEADER) | %CONN%
I'm using here document of sh to run some commands. now I want to parse the output of those commands using awk. However, everytime I execute it, I get the output of the command append with something like this "% No such child process"
This is how my script looks like.
#!/bin/sh
com = "sudo -u username /path/of/file -l"
$com <<EOF | awk '{print $0}'
Commands.
.
.
.
EOF
How am I going to use heredoc and pipeline without appending that unwanted string?
Thanks
Your variable assignment is wrong in a couple of ways. First, you aren't actually assigning a variable; you're trying to run a command named com whose arguments are = and a string "sudo ...". Spaces must not be used on either side of the =:
com="sudo ..."
Second, command lines should not be stored in a variable; the shell's parser can only make that work they way you intend for very simple commands. Type the command out in full, or use a shell function.
com () {
sudo -u username /path/to/file -l
}
com <<EOF | awk '{print $0}'
...
EOF
There's no problem, check :
$ cat <<EOF | awk '{print $1}'
a b c
1 2 3
EOF
a
1
I have a somewhat detailed query in a script that uses ? placeholders. I wanted to test this same query directly from the psql command line (outside the script). I want to avoid going in and replacing all the ? with actual values, instead I'd like to pass the arguments after the query.
Example:
SELECT *
FROM foobar
WHERE foo = ?
AND bar = ?
OR baz = ? ;
Looking for something like:
%> {select * from foobar where foo=? and bar=? or baz=? , 'foo','bar','baz' };
You can use the -v option e.g:
$ psql -v v1=12 -v v2="'Hello World'" -v v3="'2010-11-12'"
and then refer to the variables in SQL as :v1, :v2 etc:
select * from table_1 where id = :v1;
Please pay attention to how we pass string/date values using two quotes " '...' " But this way of interpolation is prone to SQL injections, because it's you who's responsible for quoting. E.g. need to include a single quote? -v v2="'don''t do this'".
A better/safer way is to let PostgreSQL handle it:
$ psql -c 'create table t (a int, b varchar, c date)'
$ echo "insert into t (a, b, c) values (:'v1', :'v2', :'v3')" \
| psql -v v1=1 -v v2="don't do this" -v v3=2022-01-01
Found out in PostgreSQL, you can PREPARE statements just like you can in a scripting language. Unfortunately, you still can't use ?, but you can use $n notation.
Using the above example:
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3 ;
EXECUTE foo('foo','bar','baz');
DEALLOCATE foo;
In psql there is a mechanism via the
\set name val
command, which is supposed to be tied to the -v name=val command-line option. Quoting is painful, In most cases it is easier to put the whole query meat inside a shell here-document.
Edit
oops, I should have said -v instead of -P (which is for formatting options) previous reply got it right.
You can also pass-in the parameters at the psql command-line, or from a batch file. The first statements gather necessary details for connecting to your database.
The final prompt asks for the constraint values, which will be used in the WHERE column IN() clause. Remember to single-quote if strings, and separate by comma:
#echo off
echo "Test for Passing Params to PGSQL"
SET server=localhost
SET /P server="Server [%server%]: "
SET database=amedatamodel
SET /P database="Database [%database%]: "
SET port=5432
SET /P port="Port [%port%]: "
SET username=postgres
SET /P username="Username [%username%]: "
SET /P bunos="Enter multiple constraint values for IN clause [%constraints%]: "
ECHO you typed %constraints%
PAUSE
REM pause
"C:\Program Files\PostgreSQL\9.0\bin\psql.exe" -h %server% -U %username% -d %database% -p %port% -e -v v1=%constraints% -f test.sql
Now in your SQL code file, add the v1 token within your WHERE clause, or anywhere else in the SQL. Note that the tokens can also be used in an open SQL statement, not just in a file. Save this as test.sql:
SELECT * FROM myTable
WHERE NOT someColumn IN (:v1);
In Windows, save the whole file as a DOS BATch file (.bat), save the test.sql in the same directory, and launch the batch file.
Thanks for Dave Page, of EnterpriseDB, for the original prompted script.
I would like to offer another answer inspired by #malcook's comment (using bash).
This option may work for you if you need to use shell variables within your query when using the -c flag. Specifically, I wanted to get the count of a table, whose name was a shell variable (which you can't pass directly when using -c).
Assume you have your shell variable
$TABLE_NAME='users'
Then you can get the results of that by using
psql -q -A -t -d databasename -c <<< echo "select count(*) from $TABLE_NAME;"
(the -q -A -t is just to print out the resulting number without additional formatting)
I will note that the echo in the here-string (the <<< operator) may not be necessary, I originally thought the quotes by themselves would be fine, maybe someone can clarify the reason for this.
It would appear that what you ask can't be done directly from the command line. You'll either have to use a user-defined function in plpgsql or call the query from a scripting language (and the latter approach makes it a bit easier to avoid SQL injection).
I've ended up using a better version of #vol7ron answer:
DO $$
BEGIN
IF NOT EXISTS(SELECT 1 FROM pg_prepared_statements WHERE name = 'foo') THEN
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3;
END IF;
END$$;
EXECUTE foo('foo','bar','baz');
This way you can always execute it in this order (the query prepared only if it does not prepared yet), repeat the execution and get the result from the last query.
I ran into space issues on my machine and therefore the sort command in unix failed because of lack of space in /tmp. In order to circumvent this, I decided to run sort with the -T option allowing it to use some other directory for creating temporary files. here is the perl script I have
my $TMPDIR = "/home/xyz/workspace/";
my $sortCommand = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T \$TMPDIR -k1,1\" }' test > test.sort";
system_call($sortCommand, "Sort");
sub system_call {
.......
}
this works perfectly on my desktop but when I run this in a different machine I get the error
"sort: cannot create temporary file: -k1,1/sortFoeXZx: No such file or directory"
Any ideas ?
You've got four levels of interpretation here (perl, shell, awk, shell). The sort shell command is
awk 'NR == 1; NR > 1 { print $0 | "sort -T $TMPDIR -k1,1" }' test > test.sort
so $TMPDIR is expanded by the inner shell. Apparently there is an environment variable of that name on your desktop machine but not in the other machine. The best fix is to make TMPDIR an environment variable and quote it properly (expanding it from Perl will cause trouble if its value contains characters that must be protected from awk or shell expansion).
$ENV{TMPDIR} = "/home/xyz/workspace/";
my $sortCommand = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T \\\"\$TMPDIR\\\" -k1,1\" }' test > test.sort";
It's strange to be relying so much on external tools from a Perl program. Ok, if sort overflows /tmp, then Perl's built-in sort is likely to run out of memory. So use Sort::External.
You have escaped the $ before the TMPDIR variable, which has resulted in $TMPDIR being passed through to the shell. As this is not defined in your environment, the command ends up as follows:
sort -T \ -k1,1
This has inadvertently escaped the following space character thus resulting in your error.
FWIW, you can some of your code by using File::Tempdir, which will let you create a temporary directory or file with the appropriate user permissions, no matter your environment.
use File::Temp 'tempdir';
my $tempdir = tempdir(CLEANUP => 1);
my $sortCommand = "awk 'NR == 1; NR > 1 { print \$0 | \"sort -T $tempdir -k1,1\" }' test > test.sort";
system_call($sortCommand, "Sort");