I was connecting to another team's Postgres DB and was trying to export data for some debugging purposes.
I noticed when I ran the below command, I got 50K rows
psql -t "sslmode=require host=MY_HOST port=MY_PORT dbname=MY_DB user=MY_USER password=MY_PASSWORD" -c "SELECT * FROM MYTABLE;" -o EXPORT.txt
$ cat EXPORT.txt | wc -l
50000
But when I log in and check the row count, it was 94K
psql -t "sslmode=require host=MY_HOST port=MY_PORT dbname=MY_DB user=MY_USER password=MY_PASSWORD"
MY_DB=> SELECT count(*) FROM MYTABLE;
94508
What is the row count for exporting the table using psql shell?
Is this a hard limit?
I wrote a bash script to loop using limit and offset as a work around.
Related
I'm trying to run the following queries in one line:
\x
select * from pg_stat_statements order by max_exec_time desc limit 10;
As follows:
kubectl -n MYNAMESPACEXXX exec -ti MYPGPOD-K8SXXX -- psql -d MY-DB -U postgres -c '\x select * from pg_stat_statements order by max_exec_time desc limit 10;'
But I get
unrecognized value "select" for "expanded"
Available values are: on, off, auto.
command terminated with exit code 1
How can we combine both \x and the SQL query ?
An alternative is to use the -c option several times:
psql -c '\x' -c 'SELECT ...'
You can also set the expanded mode from the commandline:
kubectl -n MYNAMESPACEXXX exec -ti MYPGPOD-K8SXXX -- psql -d MY-DB -U postgres --expanded -c 'select * from pg_stat_statements order by max_exec_time desc limit 10;'
I am using following psql query to connect to a remote host and split a big table into multiple csv files.
psql -h xx -p xx -U xx -d xx -c "\COPY (select * from table) TO program 'split --lines 1000' (format csv)
I am not getting what mistake I am making here.
Have you tried using STDOUT?
$ psql -d db -c "COPY (SELECT * FROM t) TO STDOUT CSV " | split -1000
Currently I can almost get a CSV from psql by simply running
psql -A -t -F'\t' -c "SELECT ...;" > myfile.csv
However it returns the number of rows at the end of the file. I can fix his with head -n -1
psql -A -t -F'\t' | head -n -1 | -c "SELECT ...;"
But with very large files seems like overkill. Is here a flag in psql where I can turn off number of records returned?
There is a number of common ways to get a CSV from PostrgeSQL (see e.g. this question). However not all of these ways are appropriate when working with Redshift, partly because Amazon Redshift is based on Postgres 8.0.2.
One can try to use --pset="footer=off" option to prevent psql from outputting the number of rows. Also, please consult 8.0.26 psql documentation.
i got this query want to to be executed remotely on my 2nd server and
#!/bin/bash
QUERY=`psql -h my2ndserverip -d testdb -U testuser 'select count(*) as result
from testdb.user where last_logged_date > (clock_timestamp() -interval '90 minutes)
echo "users = $QUERY" > tmp.txt
any tips to fix syntax ?
Use a here document (heredocuments preserve quotes AND allow shell-variable subtitution, as illustrated by the parameter 90 which is used inside single quotes):
#!/bin/bash
limit=${1:-90}
psql -h my2ndserverip -d testdb -U testuser -t -A <<EOFEOF > tmp.txt
SELECT count(*) AS result
FROM "user"
WHERE last_logged_date > clock_timestamp()-interval '${limit} minutes'
;
EOFEOF
exitcode=$?
result=`cat tmp.txt`
echo "Limit=${limit} Exitcode=${exitcode} Result=${result}"
#Eof
I suppose you want psql to omit the column headers etc, so I added the -t -A flags to the psql commandline.
BTW I changed from testdb.user, to FROM user, I don't think you have a schema named 'testdb'.
there are more than one issue
instead quotes in SQL query, you can use $$
postgres=# select interval $$90 min$$;
interval
──────────
01:30:00
(1 row)
For instance, I have a table stores value:
select * from myvalue;
val
-------
12345
(1 row)
How can I save this 12345 into a variable in postgresql or shell script?
Here's what I tried in my shell script:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
but echo $var gives me:
val ------- 12345 (1 row)
I've also tried
\set var (select * from myvalue)
in psql and when I type \set it lists:
var = '(select*frommyvalue)'
No, no, no! Use "raw data" switch from psql, like "-t" or "\t" and pipe the query to psql instead of parsing ascii-table, come on :-)
echo 'select * from myvalue;' | psql -t -h host -U user -d db
If you really need parse psql output, you could also use -H switch ( turns on HTML output ), and parse it with some perl module for parsing html tables, I used that once or twice.. Also, you may want to use a pgpass file and ~/.psqlrc for some defaults, like default DB to connect, when not specified.
psql has a -c/--command= option to accept SQL from the command line, and -t/--tuples-only option to control output formatting.
$ psql -c 'select 1+1'
?column?
----------
2
(1 row)
$ psql -t -c 'select 1+1'
2
$ VALUE=`psql -t -c 'select 1+1'`
$ echo $VALUE
2
var=`psql -Atc "select 1;"`
echo $var
1
In this answer I explain one way to do it, using a co-process to communicate back-and-forth with psql. That's overkill if all you need is to run a query and get a single result, but might be good to know if you're shell scripting with psql.
You can filter the result you get with your psql command:
var=$(psql -h host -U user -d db <<SQLSTMT
SELECT * FROM myvalue;
SQLSTMT)
var=$(cut -d' ' -f3 <<<$var)
None of these worked for me, but this did:
median_avm=psql "host=${dps1000} port=#### dbname=### user=${reduser} password=${redpass}" -c "SELECT AVG(column) FROM db.table;" -t
using a source file with ${dps1000}, ${reduser}, ${redpass} defined and manually entering port and dbname