How to check Postgres backup has been successful(Without manual efforts)? - postgresql

We have 100+ databases contains daily backups
how to check failure backups in PostgreSQL backup schedules
pg_dump -h localhost -p 5432 -U postgres -d db1 -v -f "path/file.backup"
pg_dump -h localhost -p 5432 -U postgres -d db2 -v -f "path/file.backup"
pg_dump -h localhost -p 5432 -U postgres -d db3 -v -f "path/file.backup"
pg_dump -h localhost -p 5432 -U postgres -d db4 -v -f "path/file.backup"
pg_dump -h localhost -p 5432 -U postgres -d db5 -v -f "path/file.backup"
...
like this i have 100 backup schedules

Try to do it in for loop for example?
#!/bin/bash
# create an indexed array with all your databases listed
database_names=( "1" "2" "3" )
# Declare an associative array to store dbname and dump status codes
declare -A all_dbdump_states
for db in "${database_names[#]}"; do
echo "Executing $db dump.."
pg_dump -h localhost -p 5432 -U postgres -d $db -v -f "path/file.backup"
dump_rc=$? # Save exit code of pg_dump process into variable
# After each for loop iteration, append data into array
all_dbdump_states[$db]+=$dump_rc
done
echo -e "\nListing status codes of all dumps:"
for db in "${!all_dbdump_states[#]}"; do
echo "Database [$db] status: ${all_dbdump_states[$db]}"
sleep 1
done
Here I'm echoing these pg_dump lines for better tests and made an explicit mistake in echo command to put second command fail, with exit code 127:
#!/bin/bash
# create an indexed array with all your databases listed
database_names=( "1" "2" "3" )
# Declare an assotiative array to store dbname and dump status codes
declare -A all_dbdump_states
for db in "${database_names[#]}"; do
echo "Executing $db dump.."
if [[ $db -eq 2 ]]; then
ech "pg_dump -h localhost -p 5432 -U postgres -d 2 -v -f 'path/file.backup'" &>/dev/null
dump_rc=$? # Save exit code of pg_dump process into variable
all_dbdump_states[$db]+=$dump_rc
continue
fi
echo "pg_dump -h localhost -p 5432 -U postgres -d $db -v -f 'path/file.backup'" &>/dev/null
dump_rc=$? # Save exit code of pg_dump process into variable
# After each for loop iteration, append data into array
all_dbdump_states[$db]+=$dump_rc
done
echo -e "\nListing status codes of all dumps:"
for db in "${!all_dbdump_states[#]}"; do
echo "Database [$db] status: ${all_dbdump_states[$db]}"
sleep 1
done
Sample output:
$ ./test.sh
Executing 1 dump..
Executing 2 dump..
Executing 3 dump..
Listing status codes of all dumps:
Database [1] status: 0
Database [2] status: 127
Database [3] status: 0

Related

Postgres: copy one remote DB to another

I'm trying to copy a database from one remote server to another one. I've tried several different commands from my terminal (macOS):
"pg_dump -U postgres -d [DB] -f [DB].sql"
"pg_dump -U postgres -d [DB] -h [Host] -f [DB].sql"
But nothing works. I get errors like "pg_dump: error: connection to database [DB] failed: FATAL: database [DB] does not exist".
Any ideas how to solve this problem? I've tried to edit the pg_hba.conf, but it didn't work as well..
The procedure to make this work is :
First sub-option below outputs plain text file, second custom format(binary) file
a) pg_dump -C -h host_name -U user_name -d database_name -f database.sql
-C tells pg_dump to provide the command to create the database
on restore.
b) pg_dump -Fc -h host_name -U user_name -d database_name-f database.out
To restore you need different programs
a) For plain text option 1a do:
psql -d postgres -h host_name -U user_name -f database.sql
You need to connect to existing database, postgres in this case,
and then the commands from -C above will create
the database(database_name) and then connect to it for rest of operation.
b) For custom format option 1b:
pg_restore -C -d postgres -h host_name -U user_name database.out
Note: just specify the dump file(database.out) do not use -f.
More options and details can be found:
pg_dump
and
pg_restore
Look in the Notes section at the bottom of link for examples.

create db if not exists in postgres .sh file

I want to execute these lines in the below .sh file:
#!/bin/sh +x
sudo su postgres
psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'test_db'" | grep -q 1 || psql -U postgres -c "CREATE DATABASE test_db"
But only 'sudo su postgres' is executing and the 2nd line is not executing. Can someone help me execute those 2 lines
Just pass your psql command to su:
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM pg_database WHERE datname = 'test_db'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE test_db\""
If you want to pass the datname value as a parameter, replace it by $1 in the script ($1, $2, etc. expand as the first, second, etc. parameters you pass to your script):
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM pg_database WHERE datname = '$1'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE $1\""
and call your script like this:
./script.sh test_db
You can even have optional parameters. For instance, if you want an optional parameter for the table name (pg_database in your example):
db=pg_database
if [ -n "$2" ]; then db="$2"; fi
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM $db WHERE datname = '$1'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE $1\""
and call your script like this:
./script.sh test_db
to use the default, else:
./script.sh test_db other_pg_database
I suggest you update your sudo configuration so you can directly express that you only need to as postgres and not root:
sudo -u postgres ...
For example:
user host = (postgres) command
where user is your user, host is the host name you want this apply (ALL?), and command is the name of your command (ALL?) possible prefixed with "NOPASSWD: " if you don't want to require a password.
Then do the action and deal with the error if needed instead of guarding against it:
sudo -u postgres bash -c "psql -U postgres -c "CREATE DATABASE $1"
Possible single quoting the $1 and if you want that to be robust, escape any single quotes in the database name. I showed you a shell (bash) in the above, so can easily tag on error handling.

HOW to append pg_restore restoration log to file

I have tried multiple way to append log but not able to get.
for example.
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump >db_bkp_01_10_2019.log
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump 2>db_bkp_01_10_2019.log
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump 2&>db_bkp_01_10_2019.log
I want log related to restoration like what command executed on db while restoration.
You need to use the --verbose
Try this command
./pg_restore -U postgres -p 5333 -d demodb < db_bkp_01_10_2019.dump --verbose 2>db_bkp_01_10_2019.log
Took me sometime to figure this out, it works for me in psql 11 for log:
$pg_restore -h 127.0.0.1 -U user -p 5432 -d dbname -Fc import_data.sql>import.log 2>&1

Shell Script getting terminated in the middle when unable to make DB connection

I have a function block as below which is getting called from somewhere. The call is happening fine but when it's executing any of the below psql check commands eg:
local db_availability_check_88=`psql -h 10.95.187.88 -p 5444 -c "\pset tuples_only" -c "select pg_is_in_recovery;"|grep -v "Tuples"`
The script terminates immediately if the script is unable to communicate to the database via the above psql command. I want thee script to continue its execution. If the connectivity fails it should continue with the next statements.
check_db_status_function () {
echo "entered function block db_status"
local db_availability_check_67=`psql -h 10.95.167.87 -p 5444 -c "\pset tuples_only" -c "select pg_is_in_recovery;"|grep -v "Tuples"`
local db_availability_check_68=`psql -h 10.95.167.88 -p 5444 -c "\pset tuples_only" -c "select pg_is_in_recovery;"|grep -v "Tuples"`
local db_availability_check_69=`psql -h 10.95.167.89 -p 5444 -c "\pset tuples_only" -c "select pg_is_in_recovery;"|grep -v "Tuples"`
if [[ ( $a1 = "down" ) && ( $a3 != "primary" ) && ( $db_availability_check_67 = 't' ) ]];
then
echo "pgPool services are down on the node:$a2"
echo "Promoting..."
/u01/edb/pgpool3.6/bin/pcp_attach_node -w -U pcpuser -h localhost -p 9898 $a0
fi
The script o/p is as :
Nested block 1
check_db_stat
entered function block db_status
psql.bin: could not connect to server: Connection refused
Is the server running on host "10.95.167.88" and accepting
TCP/IP connections on port 5444?
Thanks,
Sandeep

How to in insert into a specific schema in postgres using psql pipe command

I have using psql pipe command to copy a table from one database to another in Postgres. It is working fine. But I need to copy the table to a specific schema in the new database. I have gone through the documentation (used -n option for specifying schema name) but it is not working.
Command:
pg_dump -U postgres -h localhost -p 1212 -d dbname -t tablename -Ft | pg_restore -U postgres -h localhost -p 1213 -d dbname -n schemaname
you can't do it with pg_dump|pg_restore sequence. you need to alter table t set schema s; in restored db
I do it that this way:
pg_dump -U postgres -h localhost -p 1212 -d dbname -t tablename | sed "sed/oldschemaname/newschemaname/" | psql -U postgres -h localhost -p 1213 -d dbname -n schemaname
With the corresponding regular expression