I am trying to automate script to run on schedule to email query output as CSV. Following script mailing empty CSV. If we can achieve this with any other server side scripting lang also welcome, thanks
#!/bin/bash
# Export PATH in case SENDMAIL is not found by the script
export PATH=/usr/sbin:/usr/bin
# EXPORT POSTGRES SQL COMMAND AS CSV
# Adjust the Select SQL command to your liking
# Note: You may need to define the path to export.csv file
./psql -q -v ON_ERROR_STOP=1 -h pgsdbxxd-shr-01.portal.help.com -p 5432 -U username
DBname"," -c "SELECT * FROM bank.balance where Acc_Num =1234" > export.csv
# SEND AN EMAIL
# Note: You may need to define the path to export.csv file on the bottom line
( echo "to: abc#help.net"
echo "from: POSTGRESQL SERVER <abc#help.net>"
echo "Subject: PostgreSQL Export as of $(date +%F)"
echo "mime-version: 1.0"
echo "content-type: multipart/related; boundary=messageBoundary"
echo
echo "--messageBoundary"
echo "content-type: text/plain"
echo
echo "Please find the export in CSV format attached to this email."
echo "Created: $(date)."
echo
echo "--messageBoundary"
echo "content-type: text; name=export.csv"
echo "content-transfer-encoding: base64"
echo
openssl base64 < export.csv) | sendmail -t -i
Related
Using psql with COPY FROM STDIN works fine when executed via -c (inline command) but the same thing fails if -f (script file) is used. I've created a Docker-based test to demonstrate below; tested on MacOS w/ zsh and Debian w/ bash.
I was unable to find any relevant documentation on why this would be but I imagine it has to do with psql's special \copy functionality. Can someone help illuminate me?
# create test data
echo "1,apple
2,orange
3,banana">testdata.csv
# create test script
echo "drop table if exists fruits;
create table fruits (id INTEGER, name VARCHAR);
copy fruits from stdin with delimiter as ',' csv;
select * from fruits">testscript.pg
# create network
docker network create pgtest
# run Postgres server
echo "starting postgres server"
PG_CONTAINER_ID=$(docker run -d --name=pgtest --rm --network=pgtest -h database -e POSTGRES_USER=user1 -e POSTGRES_PASSWORD=pass1 -e POSTGRES_DB=db1 -p 6432:5432 postgres:12)
echo "sleeping for 5 seconds (wait for server to start)"
sleep 5
docker logs $PG_CONTAINER_ID
echo "*"
echo "*"
echo "*"
echo "run psql script using inline with -c"
cat testdata.csv | docker run -i --rm --network=pgtest postgres:12 psql postgres://user1:pass1#database:5432/db1 -c "$(cat testscript.pg)"
echo "*"
echo "*"
echo "*"
echo "run psql script using file with -f"
cat testdata.csv | docker run -i -v $PWD:/host --rm --network=pgtest postgres:12 psql postgres://user1:pass1#database:5432/db1 -f /host/testscript.pg
# stop server
echo "*"
echo "*"
echo "*"
docker stop $PG_CONTAINER_ID
docker rm $PG_CONTAINER_ID
The output of the psql commands look like this:
*
*
*
run psql script using inline with -c
NOTICE: table "fruits" does not exist, skipping
id | name
----+--------
1 | apple
2 | orange
3 | banana
(3 rows)
*
*
*
run psql script using file with -f
DROP TABLE
CREATE TABLE
psql:/host/testscript.pg:5: ERROR: invalid input syntax for type integer: "select * from fruits"
CONTEXT: COPY fruits, line 1, column id: "select * from fruits"
In the first case, (execution with -c), the copy data are read from standard input.
In the second case (execution with -f), the input file acts as input to psql (if you want, standard input is redirected from that file). So PostgreSQL interprets the rest of the file as COPY data and complains about the content. You'd have to mix the COPY data in with the file:
/* script with copy data */
COPY mytable FROM STDIN (FORMAT 'csv');
1,item 1,2021-11-01
2,"item 2, better",2021-11-11
\.
/* next statement */
ALTER TABLE mytable ADD newcol text;
I am trying to check a table and send email based on the number of records. If the records are zero for the particular date then success email shall go out. If there is any record for the particular day then error messages should go out. My problem is - the opposite is happening. even though there are no records, error mail is going out.
Below is the script
#!/bin/bash
export PGPASSWORD=xyz
TIMESTAMP=`date "+%Y-%m-%d %H:%M:%S"`
mailid='xyz#gmail.com'
cd /opt/postgres/pgsql/bin
vartest=`./psql -U user -h host -p port umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"` >> /opt/rmapp/test_error.log
if [$vartest -eq 0]
then
echo 'Files processed without errors' | mutt -s "Processing Success `date`" $mailid $attached >> /opt/rmapp/test_error.log
else
error=`./psql -U user -h host -p port umrm-t -c "select count (*) from dataprocess_errors where error_date = current_date and error_message like '%More than 25 employees%';"` >> /opt/rmapp/test_error.log
if [$error -eq 0]
then
echo 'Please check the table and fix the errorneous records' | mutt -s "Alert!! $vartest records failed in processing `date`" $mailid $attached >> /opt/rmapp/test_error.log
else
echo 'Please check the table' | mutt -s "Alert!!! processing stopped `date`" $mailid $attached >> /opt/rmapp/test_error.log
fi
fi
echo "Error Check completed at : $TIMESTAMP" >> /opt/rmapp/test_error.log
the second if condition in the else section is to identify the different kind of errors based on the output of variable "error".
I am a noob in shell scripting so kindly help.
Thanks in advance for all the help.
Considering your psql command is working fine, you have only one issue in above script. In both if conditions space after '[' and before ']' is missing.
So your if conditions should be like :
if [ $vartest -eq 0 ]
and
if [ $error -eq 0 ]
In your lines
vartest=`./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"`
and
error=`./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm-t -c "sele ........
you are getting the standard out from the command - not the error code
to get the error code you can (as an example):
./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"
if [ $? -ne 0 ];
then
echo "there was an error on above command"
fi
note: the if must follow directly after the command you are testing the error code on
REF: https://en.wikipedia.org/wiki/Bash_(Unix_shell)
I have created a bash script to copy CSV data into a Postgres table.
#!/bin/sh
PSQL=/Library/PostgreSQL/13/bin/psql
DB_USER=user
DB_HOST=localhost
DB_NAME=dbname
export PGPASSWORD="password"
file_path="/Users/user/test.csv"
echo "copy started .."
result=$($PSQL -X -U $DB_USER -h $DB_HOST -P t -P format=unaligned -c "\\COPY table_name FROM '$file_path' DELIMITER ',' CSV HEADER" $DB_NAME)
echo "copy completed : $result"
This always returns COPY 0 and no records are added.
Tried running the copy command on terminal as well but same result.
I am missing something here?
I am trying to store a Greenplum query result into a csv but it is storing only one row. I want to store multiple rows. Can anyone help with this.
I tried the below already but it is storing only one row.
r= `psql -A -t -q -h $hostname -U $username -d $dbname <<-THE_END
COPY(select * From ${gschema}.${gtable} order by ${key} limit 3) TO STDOUT with NULL as ' '
THE_END` > /home/gp.csv
I also tried below which is storing the result but I am unable to pass table name as parameter in the below query. For standard table names the output is as desired.
psql -h $hostname -U $username -d $dbname -c "COPY (select * from table_name order by key limit 3) TO STDOUT "> /home/gp.csv
Can anyone help me with this please.
please note that I am trying to embed the above in a shell script
Try this:
psql -A -t -q -h $hostname -U $username -d $dbname -c "select * from $gschema.$gtable order by $key limit 3" > /home/gp.csv
You could also put the SQL in a file:
example.sql
select * from :gschema.:gtable order by :key limit 3
And then to use the sql file:
psql -A -t -q -h $hostname -U $username -d $dbname -f example.sql -v gschema=public -v gtable=customer -v key=id > /home/gp.csv
I'm trying to create an automated backup in Postgresql using below link, but I don't know where to find the needed dll, I'm stuck here. Can't proceed to next instruction because of this. Can anyone knows how to do it? Need help please.
https://wiki.postgresql.org/wiki/Automated_Backup_on_Windows
comerr32.dll
gssapi32.dll
k5sprt32.dll
krb_32.dll
libeay32.dll
libiconv2.dll
libpq.dll
Microsoft.VC80.CRT.manifest
msvcm80.dll
msvcp80.dll
msvcr80.dll
pg_dump.dll
ssleay32.dll
zlib1.dll
Here's batch file script:
#echo off
for /f "tokens=1-4 delims=/ " %%i in ("%date%") do (
set dow=%%i
set month=%%j
set day=%%k
set year=%%l
)
set datestr=%month%_%day%_%year%
echo datestr is %datestr%
set BACKUP_FILE=<backup_name_>_%datestr%.backup
echo backup file name is %BACKUP_FILE%
SET PGPASSWORD=<password>
echo on
bin\pg_dump -i -h <localhost> -p 5432 -U <postgres> -F c -b -v -f %BACKUP_FILE% <db_name>
Is there missing syntax?
When manually executed error shows:
bin\pg_dump: illegal option -- i
I execute something like this
pg_dump.exe -h %SERVER% -p 5432 -U postgres -Fc -d %basedatos% -v -f %filebackup%
This variables replace value respective.