I have a script that I run nightly in cron to backup some postgres databases for several hosts on my network. I have a way of getting alerted that the script fails by leveraging the exit status, but it doesn't tell me WHY it failed.
Based off the following code, how can I capture any errors that occur when the script is run, and email them to me so I can have a better understanding of what happened.
FILEDATE=`date '+%Y-%m-%d'`
BASEDIR=/u1/$1/db_dumps
PGDUMP=/path/to/pg_dump
HOST=$1
DB=$2
if [ $DB == all ]
then
for ALLDUMPS in db1 db2 db3
do
ssh root#$HOST "env PGUSER=pguser PGPASSWORD=pgpassword $PGDUMP -Fc $ALLDUMPS" | pbzip2 > $BASEDIR/$FILEDATE-$HOST-$ALLDUMPS.dump.bz2
if [ $? -ne 0 ]
then mutt -s "dbdumper could not create a backup of $ALLDUMP from $HOST" me#myemail.com < /dev/null
fi
done
else
ssh root#$HOST "env PGUSER=pguser PGPASSWORD=pgpassword $PGDUMP -Fc $DB" | pbzip2 > $BASEDIR/$FILEDATE-$HOST-$DB.dump.bz2
if [ $? -ne 0 ]
then mutt -s "dbdumper failed to create a backup of $DB from $HOST" me#myemail.com < /dev/null
fi
fi
Capture stderr from the ssh command, and email that to yourself.
stderr=$( ssh root#$HOST "env PGUSER=user PGPASSWORD=pw $PGDUMP -Fc $ALLDUMPS" |
pbzip2 2>&1 > "$BASEDIR/$FILEDATE-$HOST-$ALLDUMPS.dump.bz2" )
if [ $? -ne 0 ]; then
subj="dbdumper could not create a backup of $ALLDUMP from $HOST"
mutt -s "$subj" me#myemail.com <<< "$stderr"
fi
Related
Been googling around for this but I can't get the exact answer.
I am building a mobile app and I want to run additional migration scripts when the environment is "local".
I have a docker-compose-local.yml which builds the db
database:
build:
context: ./src/Database/
dockerfile: Dockerfile
container_name: database
ports:
- 1401:1433
environment:
ACCEPT_EULA: 'Y'
SA_PASSWORD: 'password'
ENVIRONMENT: 'local'
networks:
- my-network
and then a Dockerfile with an entrypoint
ENTRYPOINT ["/usr/src/app/entry-point.sh"]
And then a script that runs migrations.
#!/bin/bash
# wait for MSSQL server to start
export STATUS=1
i=0
MIGRATIONS=$(ls migrations/*.sql | sort -V)
SEEDS=$(ls seed/*.sql)
while [[ $STATUS -ne 0 ]] && [[ $i -lt 30 ]]; do
i=$i+1
/opt/mssql-tools/bin/sqlcmd -t 1 -U sa -P $SA_PASSWORD -Q "select 1" >> /dev/null
STATUS=$?
done
if [ $STATUS -ne 0 ]; then
echo "Error: MSSQL SERVER took more than thirty seconds to start up."
exit 1
fi
echo "======= MSSQL SERVER STARTED ========" | tee -a ./config.log
# Run the setup script to create the DB and the schema in the DB
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i create-database.sql
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i create-database-user.sql
for f in $MIGRATIONS
do
echo "Processing migration $f"
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i $f
done
# RUN THIS ONLY FOR ENVIRONMENT = local
for s in $SEEDS
do
echo "Seeding $s"
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i $s
done
Currently everything works perfectly fine, except the seeds are added for all environments.
I only want to run the seed scripts if environment = local.
How can this condition be written into this script?
Alternative is there a cleaner way to do this?
Thanks
There are multiple ways to achieve your goal. Three that come to mind quickly are:
Insert and check the environment variable in the script (What you are trying to do now)
Have two versions of the script in the Container and change the entrypoint in the docker-compose file (either with environment variables or by using multiple compose files)
Build two different versions of the docker image for local and production environment
With your current setup the first alternative is the easiest:
# RUN THIS ONLY FOR ENVIRONMENT = local
if [[ "$ENVIRONMENT" == "local" ]]; then
for s in $SEEDS
do
echo "Seeding $s"
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i $s
done
fi
I am trying to check a table and send email based on the number of records. If the records are zero for the particular date then success email shall go out. If there is any record for the particular day then error messages should go out. My problem is - the opposite is happening. even though there are no records, error mail is going out.
Below is the script
#!/bin/bash
export PGPASSWORD=xyz
TIMESTAMP=`date "+%Y-%m-%d %H:%M:%S"`
mailid='xyz#gmail.com'
cd /opt/postgres/pgsql/bin
vartest=`./psql -U user -h host -p port umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"` >> /opt/rmapp/test_error.log
if [$vartest -eq 0]
then
echo 'Files processed without errors' | mutt -s "Processing Success `date`" $mailid $attached >> /opt/rmapp/test_error.log
else
error=`./psql -U user -h host -p port umrm-t -c "select count (*) from dataprocess_errors where error_date = current_date and error_message like '%More than 25 employees%';"` >> /opt/rmapp/test_error.log
if [$error -eq 0]
then
echo 'Please check the table and fix the errorneous records' | mutt -s "Alert!! $vartest records failed in processing `date`" $mailid $attached >> /opt/rmapp/test_error.log
else
echo 'Please check the table' | mutt -s "Alert!!! processing stopped `date`" $mailid $attached >> /opt/rmapp/test_error.log
fi
fi
echo "Error Check completed at : $TIMESTAMP" >> /opt/rmapp/test_error.log
the second if condition in the else section is to identify the different kind of errors based on the output of variable "error".
I am a noob in shell scripting so kindly help.
Thanks in advance for all the help.
Considering your psql command is working fine, you have only one issue in above script. In both if conditions space after '[' and before ']' is missing.
So your if conditions should be like :
if [ $vartest -eq 0 ]
and
if [ $error -eq 0 ]
In your lines
vartest=`./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"`
and
error=`./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm-t -c "sele ........
you are getting the standard out from the command - not the error code
to get the error code you can (as an example):
./psql -U rmdbprd -h atla-db-rmgr-prd-priv.com -p 5432 umrm -t -c "select count(*) from dataprocess_errors where error_date = current_date"
if [ $? -ne 0 ];
then
echo "there was an error on above command"
fi
note: the if must follow directly after the command you are testing the error code on
REF: https://en.wikipedia.org/wiki/Bash_(Unix_shell)
I Used Common Postgresql backup script from Automated_Backup_on_Linux:
#!/bin/bash
if [ ! $HOSTNAME ]; then HOSTNAME="localhost"; fi
if [ ! $USERNAME ]; then USERNAME="postgres"; fi
BACKUP_DIRECTORY="/Users/xeranta/Documents/AW/"
CURRENT_DATE=$(date "+%Y%m%d")
if [ -z "$1" ]; then
pg_dump -U postgres -h localhost -p 5432 db_gocampus_unmul \
> $BACKUP_DIRECTORY/db_gocampus_unmul.sql
else
pg_dump $1 | gzip - > $BACKUP_DIRECTORY/$1_$CURRENT_DATE.sql.gz
fi
It runs In terminal
$ ~/Documents/AW/./dbbackup.sh
But does not runs when in set this in CRONTAB in MacOS Sierra version 10.12.6
I have this error
/Users/xeranta/Documents/AW/./backup.sh: line 36: pg_dumpall: command
not found
The cause of the problem is that pg_dump is on your PATH in the interactive shell, but not in the cron job.
You should use an absolute path similar to this:
PGPATH=/wherever/your/postgres/binaries/are
"$PGPATH"/pg_dump ...
I am trying to adapt a shell script set made for running on Debian 7 to work on Ubuntu 16.
I got to change successfully all except a part that executes PosgreSQL database commands.
Former version of script has these lines:
service postgresql restart
psql -q -U postgres -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -q -U postgres -c "CREATE DATABASE db_crm" -o $log &> /dev/null
When I tried to run psql as above on Ubuntu 16, OS didn't recognize command. It is important to say that script is called with sudo.
I got to find a way to run only database script on Ubuntu 16 changing code so:
service postgresql restart
su postgres <<EOF
psql -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -c "CREATE DATABASE db_crm" -o $log &> /dev/null
However, this same script doesn't work when it is called by main script. Following messages are presented:
here-document at line 41 delimited by end-of-file (wanted 'EOF')
syntax error: unexpected end of file
Even replacing EOF to beggining of next line, error continues.
If there is a way to use psql in shell script without to use EOF would be better.
The reason your script is failing, is you forgot the EOF at the end of input.
su postgres <<EOF
psql -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -c "CREATE DATABASE db_crm" -o $log &> /dev/null
EOF #<<<--- HERE
An easy way to do this is to put your commands into a temporary file, then re-direct that into psql. Obviously you don't want this to stop for a password prompt, in this case either use a user that doesn't need it, or set $PGPASSWORD - or prompt at the beginning of the script - there's lots of ways around.
#! /usr/bin/env bash
# PGPASSWORD='' #(set this to stop password prompts, but it's insecure)
PSQL="psql -q -U postgres -o $log" #TODO define $log
TMPFILE="/tmp/sql-tmp.`date +%Y%m%d_%H%M%S_%N`.sql"
# TODO - check $TMPFILE does not exist already
echo "DROP DATABASE IF EXISTS db_crm;" > "$TMPFILE"
echo "CREATE DATABASE db_cr;" >> "$TMPFILE"
# ... etc.
# run the command, throw away any stdout, stderr
PSQL < "$TMPFILE" 2>&1 > /dev/null
# exit with the psql error code
err_code=$?
exit $?
I need to use mongodb with the --rest option. But mongodb is started automatically on boot, so I guess I need to modify a file or something.
Where can I add this --rest option?
I have this file at /etc/init/mongodb.conf, not sure what to edit:
# Ubuntu upstart file at /etc/init/mongodb.conf
limit nofile 20000 20000
kill timeout 300 # wait 300s between SIGTERM and SIGKILL.
pre-start script
mkdir -p /var/lib/mongodb/
mkdir -p /var/log/mongodb/
end script
start on runlevel [2345]
stop on runlevel [06]
script
ENABLE_MONGODB="yes"
if [ -f /etc/default/mongodb ]; then . /etc/default/mongodb; fi
if [ "x$ENABLE_MONGODB" = "xyes" ]; then exec start-stop-daemon --start --quiet --chuid mongodb --exec /usr/bin/mongod -- --config /etc/mongodb.conf; fi
end script
And this file at /etc/init.d/mongodb:
#!/bin/sh -e
# upstart-job
#
# Symlink target for initscripts that have been converted to Upstart.
set -e
INITSCRIPT="$(basename "$0")"
JOB="${INITSCRIPT%.sh}"
if [ "$JOB" = "upstart-job" ]; then
if [ -z "$1" ]; then
echo "Usage: upstart-job JOB COMMAND" 1>&2
exit 1
fi
JOB="$1"
INITSCRIPT="$1"
shift
else
if [ -z "$1" ]; then
echo "Usage: $0 COMMAND" 1>&2
exit 1
fi
fi
COMMAND="$1"
shift
if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]; then
ECHO=echo
else
ECHO=:
fi
$ECHO "Rather than invoking init scripts through /etc/init.d, use the service(8)"
$ECHO "utility, e.g. service $INITSCRIPT $COMMAND"
# Only check if jobs are disabled if the currently _running_ version of
# Upstart (which may be older than the latest _installed_ version)
# supports such a query.
#
# This check is necessary to handle the scenario when upgrading from a
# release without the 'show-config' command (introduced in
# Upstart for Ubuntu version 0.9.7) since without this check, all
# installed packages with associated Upstart jobs would be considered
# disabled.
#
# Once Upstart can maintain state on re-exec, this change can be
# dropped (since the currently running version of Upstart will always
# match the latest installed version).
UPSTART_VERSION_RUNNING=$(initctl version|awk '{print $3}'|tr -d ')')
if dpkg --compare-versions "$UPSTART_VERSION_RUNNING" ge 0.9.7
then
initctl show-config -e "$JOB"|grep -q '^ start on' || DISABLED=1
fi
case $COMMAND in
status)
$ECHO
$ECHO "Since the script you are attempting to invoke has been converted to an"
$ECHO "Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB"
$COMMAND "$JOB"
;;
start|stop)
$ECHO
$ECHO "Since the script you are attempting to invoke has been converted to an"
$ECHO "Upstart job, you may also use the $COMMAND(8) utility, e.g. $COMMAND $JOB"
if status "$JOB" 2>/dev/null | grep -q ' start/'; then
RUNNING=1
fi
if [ -z "$RUNNING" ] && [ "$COMMAND" = "stop" ]; then
exit 0
elif [ -n "$RUNNING" ] && [ "$COMMAND" = "start" ]; then
exit 0
elif [ -n "$DISABLED" ] && [ "$COMMAND" = "start" ]; then
exit 0
fi
$COMMAND "$JOB"
;;
restart)
$ECHO
$ECHO "Since the script you are attempting to invoke has been converted to an"
$ECHO "Upstart job, you may also use the stop(8) and then start(8) utilities,"
$ECHO "e.g. stop $JOB ; start $JOB. The restart(8) utility is also available."
if status "$JOB" 2>/dev/null | grep -q ' start/'; then
RUNNING=1
fi
if [ -n "$RUNNING" ] ; then
stop "$JOB"
fi
# If the job is disabled and is not currently running, the job is
# not restarted. However, if the job is disabled but has been forced into the
# running state, we *do* stop and restart it since this is expected behaviour
# for the admin who forced the start.
if [ -n "$DISABLED" ] && [ -z "$RUNNING" ]; then
exit 0
fi
start "$JOB"
;;
reload|force-reload)
$ECHO
$ECHO "Since the script you are attempting to invoke has been converted to an"
$ECHO "Upstart job, you may also use the reload(8) utility, e.g. reload $JOB"
reload "$JOB"
;;
*)
$ECHO
$ECHO "The script you are attempting to invoke has been converted to an Upstart" 1>&2
$ECHO "job, but $COMMAND is not supported for Upstart jobs." 1>&2
exit 1
esac
It's probably cleaner to enable the REST interface via /etc/mongodb.conf by adding a line of:
rest = true
That setting is documented here.
MongoDB version 2.6 has switched to a YAML config file. The following two entries are required to prevent the following startup warning:
mongodb WARNING: --rest is specified without --httpinterface
net:
http:
enabled: true
RESTInterfaceEnabled: true
When u start the server using command mongod , add --rest option with command mongod like this mongod --rest.
refer mongod - MongoDB Manual 2.6.
After run command complete , u can use the following the simple Restful API:
http://127.0.0.1:28017/databaseName/collectionName/
Here is simple RestFul API Doc.
Just start the server using mongod --rest
Note: By default, the rest API's are inaccessible due to security issues. The web interface is accessible at localhost:<port>, where the number is 1000 more than the mongod port. For example, your mongodb server is running at 27017 (by default) then you can access mongodb at
http://127.0.0.1:28017/<db-name>/<collection-name>/