Using shell script store PostgreSQL query on a variable - postgresql

I want to store following postgreSQL query result in a variable. I am writing command on the shell script.
psql -p $port -c "select pg_relation_size ('tableName')" postgres
I need variable to save the result on a file. I have tried following but it is not working
var= 'psql -p $port -c "select pg_relation_size ('tableName')" '

Use a shell HERE document like:
#!/bin/sh
COUNT=`psql -A -t -q -U username mydb << THE_END
SELECT count (DISTINCT topic_id) AS the_count
FROM react
THE_END`
echo COUNT=${COUNT}
The whole psql <<the_end ... stuff here ... the_end statement is packed into backticks
the output of the execution of the statement inside the backticks is used as a value for the COUNT shell variable
The -A -t -q are needed to suppress column headers and error output
inside a here document, shell variable substitution works, even in single quotes!
So, you could even do:
#!/bin/sh
DB_NAME="my_db"
USR_NAME="my_name"
TBL_NAME="my_table"
COL_NAME="my_column"
COUNT=`psql -A -t -q -U ${USR_NAME} ${DB_NAME} << THE_END
SELECT COUNT(DISTINCT ${COL_NAME} ) AS the_count
FROM ${TBL_NAME}
THE_END`
echo COUNT=${COUNT}

to run a query inline you have to wrap it in grave accents, not single quotes:
$ vim `which fancyexecfileinpath`
psql lets you run queries from command line, but I guess you should be inputting complete information. you might be missing the database name.
postgres#slovenia:~$ psql -d mydbname -c "select * from applications_application;"
postgres#slovenia:~$ specialvar=`psql -d flango -c "select * from applications_application;"`
postgres#slovenia:~$ echo $specialvar
id | name | entities | folder | def_lang_id | ... | 2013-07-09 15:16:57.33656+02 | /img/app3.png (1 row)
postgres#slovenia:~$
notice the grave accents when assigning it to specialvar
otherwise you'll be setting it to a string.
There shouldn't be any space between the variable and the equals sign ("=") and the value ( http://genepath.med.harvard.edu/mw/Bash:HOW_TO:_Set_an_environment_variable_in_the_bash_shell )

Related

Why does psql -f COPY FROM STDIN fail when -c succeeds?

Using psql with COPY FROM STDIN works fine when executed via -c (inline command) but the same thing fails if -f (script file) is used. I've created a Docker-based test to demonstrate below; tested on MacOS w/ zsh and Debian w/ bash.
I was unable to find any relevant documentation on why this would be but I imagine it has to do with psql's special \copy functionality. Can someone help illuminate me?
# create test data
echo "1,apple
2,orange
3,banana">testdata.csv
# create test script
echo "drop table if exists fruits;
create table fruits (id INTEGER, name VARCHAR);
copy fruits from stdin with delimiter as ',' csv;
select * from fruits">testscript.pg
# create network
docker network create pgtest
# run Postgres server
echo "starting postgres server"
PG_CONTAINER_ID=$(docker run -d --name=pgtest --rm --network=pgtest -h database -e POSTGRES_USER=user1 -e POSTGRES_PASSWORD=pass1 -e POSTGRES_DB=db1 -p 6432:5432 postgres:12)
echo "sleeping for 5 seconds (wait for server to start)"
sleep 5
docker logs $PG_CONTAINER_ID
echo "*"
echo "*"
echo "*"
echo "run psql script using inline with -c"
cat testdata.csv | docker run -i --rm --network=pgtest postgres:12 psql postgres://user1:pass1#database:5432/db1 -c "$(cat testscript.pg)"
echo "*"
echo "*"
echo "*"
echo "run psql script using file with -f"
cat testdata.csv | docker run -i -v $PWD:/host --rm --network=pgtest postgres:12 psql postgres://user1:pass1#database:5432/db1 -f /host/testscript.pg
# stop server
echo "*"
echo "*"
echo "*"
docker stop $PG_CONTAINER_ID
docker rm $PG_CONTAINER_ID
The output of the psql commands look like this:
*
*
*
run psql script using inline with -c
NOTICE: table "fruits" does not exist, skipping
id | name
----+--------
1 | apple
2 | orange
3 | banana
(3 rows)
*
*
*
run psql script using file with -f
DROP TABLE
CREATE TABLE
psql:/host/testscript.pg:5: ERROR: invalid input syntax for type integer: "select * from fruits"
CONTEXT: COPY fruits, line 1, column id: "select * from fruits"
In the first case, (execution with -c), the copy data are read from standard input.
In the second case (execution with -f), the input file acts as input to psql (if you want, standard input is redirected from that file). So PostgreSQL interprets the rest of the file as COPY data and complains about the content. You'd have to mix the COPY data in with the file:
/* script with copy data */
COPY mytable FROM STDIN (FORMAT 'csv');
1,item 1,2021-11-01
2,"item 2, better",2021-11-11
\.
/* next statement */
ALTER TABLE mytable ADD newcol text;

How to return a value from psql to bash and use it?

Suppose I created a sequence in postgresql:
CREATE SEQUENCE my_seq;
I store the below line in an sql file get_seq.sql
SELECT last_value FROM my_seq;
$SUDO psql -q -d database_bame -f get_seq.sql
How do I get the int number returned by SELECT into bash and use it?
You can capture the result of a command using the VAR=$(command) syntax:
VALUE=$(psql -qtAX -d database_name -f get_seq.sql)
echo $VALUE
The required psql options mean:
-t only tuple
-A output not unaligned
-q quiet
-X Don't run .psqlrc file
Try:
LAST_VALUE=`echo "SELECT last_value FROM my_seq;" | psql -qAt -d database_name`

psql - write a query and the query's output to a file

In postgresql 9.3.1, when interactively developing a query using the psql command, the end result is sometimes to write the query results to a file:
boron.production=> \o /tmp/output
boron.production=> select 1;
boron.production=> \o
boron.production=> \q
$ cat /tmp/output
?column?
----------
1
(1 row)
This works fine. But how can I get the query itself to be written to the file along with the query results?
I've tried giving psql the --echo-queries switch:
-e, --echo-queries
Copy all SQL commands sent to the server to standard output as well.
This is equivalent to setting the variable ECHO to queries.
But this always echoes to stdout, not to the file I gave with the \o command.
I've tried the --echo-all switch as well, but it does not appear to echo interactive input.
Using command editing, I can repeat the query with \qecho in front of it. That works, but is tedious.
Is there any way to direct an interactive psql session to write both the query and the query output to a file?
You can try redirecting the stdout to a file directly from your shell (Win or Linux should work)
psql -U postgres -c "select 1 as result" -e nomedb >> hello.txt
This has the drawback of not letting you see the output interactively. If that's a problem, you can either tail the output file in a separate terminal, or, if in *nix, use the tee utility:
psql -U postgres -c "select 1 as result" -e nomedb | tee hello.txt
Hope this helps!
Luca
I know this is an old question, but at least in 9.3 and current versions this is possible using Query Buffer meta-commands shown in the documentation or \? from the psql console: https://www.postgresql.org/docs/9.3/static/app-psql.html
\w or \write filename
\w or \write |command
Outputs the current query buffer to the file filename or pipes it to the shell command command.
Please try this format as I got the output from the same:
psql -h $host -p $port -q -U $user -d $Dbname -c "SELECT \"Employee-id\",\"Employee-name\" FROM Employee_table" >> Employee_Date.csv
I need the output in a CSV file.

Making an empty output from psql

I am running a psql command that executes a complex query. There's nothing that query produces, as such, psql returns "(No rows)" in the output.
Is there a way to make psql to return an empty string?
I've tried using --pset=tuples-only=on and --pset=footer=off and -q in all variations, and it doesn't seem to work.
Footer option works while in psql shell prompt, but doesn't work from script.
Tried on 9.1.7, need this for 8.4, 9.1 and 9.2.
May be good enough:
$ psql -Axt -c 'select 1 where 1=0'
produces an empty string
EDIT following comments:
The command above produces an empty line, so that includes and end-of-line.
To produce nothing at all, remove the -x option.
Not that it would make any difference to the shell anyway, as shown below:
with -x:
r=`psql -Atx -d test -c "select 1 where 1=0"`
echo $r | od -c
0000000 \n
0000001
without -x:
r=`psql -At -d test -c "select 1 where 1=0"`
echo $r | od -c
0000000 \n
0000001
Is there a way to make psql to return an empty string?
You can simply append SELECT '' after the first query.
Ex.:
psql -Atp5432 mydb -c "SELECT 1 WHERE FALSE; SELECT ''"
Replace SELECT 1 WHERE FALSE with your complex query that doesn't return a row.

Store PostgreSQL query result to Shell or PostgreSQL variable

For instance, I have a table stores value:
select * from myvalue;
val
-------
12345
(1 row)
How can I save this 12345 into a variable in postgresql or shell script?
Here's what I tried in my shell script:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
but echo $var gives me:
val ------- 12345 (1 row)
I've also tried
\set var (select * from myvalue)
in psql and when I type \set it lists:
var = '(select*frommyvalue)'
No, no, no! Use "raw data" switch from psql, like "-t" or "\t" and pipe the query to psql instead of parsing ascii-table, come on :-)
echo 'select * from myvalue;' | psql -t -h host -U user -d db
If you really need parse psql output, you could also use -H switch ( turns on HTML output ), and parse it with some perl module for parsing html tables, I used that once or twice.. Also, you may want to use a pgpass file and ~/.psqlrc for some defaults, like default DB to connect, when not specified.
psql has a -c/--command= option to accept SQL from the command line, and -t/--tuples-only option to control output formatting.
$ psql -c 'select 1+1'
?column?
----------
2
(1 row)
$ psql -t -c 'select 1+1'
2
$ VALUE=`psql -t -c 'select 1+1'`
$ echo $VALUE
2
var=`psql -Atc "select 1;"`
echo $var
1
In this answer I explain one way to do it, using a co-process to communicate back-and-forth with psql. That's overkill if all you need is to run a query and get a single result, but might be good to know if you're shell scripting with psql.
You can filter the result you get with your psql command:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
var=$(cut -d' ' -f3 <<<$var)
None of these worked for me, but this did:
median_avm=psql "host=${dps1000} port=#### dbname=### user=${reduser} password=${redpass}" -c "SELECT AVG(column) FROM db.table;" -t
using a source file with ${dps1000}, ${reduser}, ${redpass} defined and manually entering port and dbname