I'm using a Powershell script to start a Docker container with Postgres:
docker run -p ${host_port}:${remote_port} --name $container_name -d $database_name
# 0b. Wait for the container and the postgres database to be ready
Do
{
echo "Waiting for database system to start up..."
$timeout++
sleep 1
} until ((docker exec $container_name psql --username=$database_user_name --dbname=$database_name --command="SELECT 1;") -Or ($timeout -eq $timeout_limit))
if ($timeout -eq $timeout_limit)
{
Throw "Database system failed to start up."
exit
}
else {
# Do stuff
}
An issue I ran into was that the database was not always ready by the time I imported my schema. I added the Do-Until loop to continue "pinging" the database until it gets a response, or until 10 seconds have passed.
This works okay. And this is the console output:
Waiting for database system to start up...
psql: FATAL: the database system is starting up
Waiting for database system to start up...
Is there any way to prevent that second line from appearing?
psql: FATAL: the database system is starting up
I've tried to redirect the "pinging" to /dev/null but this fails when executing it like so,
docker exec $container_name psql --username=$database_user_name --dbname=$database_name --command="SELECT 1;" > /dev/null
I guess you have to redirect Stderr too. How does 2>&1 instead of just > work?
Related
I'm writing a bash script that backs up a mongodb instance using mongodump. There are multiple steps to the script, that only run if the dump is successful, so I need an error code that tells me if the backup ran successfully. I've been using the following:
for i in $(seq 1 30); do
mongodump --host mongodb -u user -p password --archive | gzip > backup.gz
check=$?
echo "$check"
if [ "$check" -eq "0" ]; then
break
fi
done
if [ "$check" -eq "0" ]; then
echo "do something with the file"
fi
This works fine, when mongodump is actually successful, but the issue is, even when mongodump fails, it returns 0. Which seems counter to what I understand from their documentation here.
For example, if I disconnect this server from the database by unplugging a network cable, it fails with the error Failed: can't create session: could not connect to server: server selection error: server selection timeout. But the return code for that is also 0, just like a success.
Maybe it's supposed to be, but how do I error check for network errors if the error code for network errors is the same as success?
Or is that success coming from the gzip? Which would make my question, why does mongodump create the backup file even on network failure?
Replace $? with "${PIPESTATUS[0]}" to get exit status of first command (mongodump ...) in your pipe.
From man bash:
PIPESTATUS: An array variable containing a list of exit status values from the processes in the most-recently-executed foreground pipeline (which may contain only a single command).
Or avoid a pipe. Replace
mongodump --host mongodb -u user -p password --archive | gzip > backup.gz
with
mongodump --host mongodb -u user -p password --archive > >(gzip > backup.gz)
I have an existing jenkins job that kicks off a shell script to copy my prod environment into qa.
We added a lot of data to prod (gzip dump went from 2gig to 15gig) and all of the sudden my jenkins jobs started failing.
We are running postgres 9.5 in aws and jenkins 2.171. all jenkins jobs are executed on master which is the same server with 6 executors. There are no memory/cpu/disk space issues
Tried a few things: statement_timeout on the postgres instance is already 0. Switching from bash to sh for some reason helped on some scripts but not others. In particular this one is still having various psql statements Killed. the script works fine when run from an interactive shell.
Also tried disabling Process Tree Killer https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller. no go.
Here's the code from two of the more innocuous commands that should run pretty quickly. $POSTGRES_HOST_OPTS only has the db name and port:
echo -e "Running POSTGIS command"
psql $POSTGRES_HOST_OPTS -U $POSTGRES_ENV_POSTGRES_USER_PROD -d postgres -c "CREATE EXTENSION postgis;"
echo -e "Creating temporary user dv3_qa_tmp so we can rename the $POSTGRES_ENV_POSTGRES_USER_PROD user\n"
psql $POSTGRES_HOST_OPTS -U $POSTGRES_ENV_POSTGRES_USER_PROD -d postgres -c "create role dv3_qa_tmp password '$PGPASSWORD_QA' createdb createrole inherit login;"
Here's the output from jenkins console:
Waiting for new instance to be available...
-e Renaming database dv3_prod to dv3_qa
Killed
-e Running POSTGIS command
Killed
-e Creating temporary user dv3_qa_tmp so we can rename the dv3_prod_user user
Killed
-e Renaming user dv3_prod_user to dv3_qa_user
Killed
Killed
-e
All done
From the jenkins.log there is something on file descriptors but not sure how that is related. I've also tried redirecting stderr which gets rid of this message but doesn't stop the commands being killed.
Apr 10, 2019 4:23:31 PM hudson.Proc$LocalProc join
WARNING: Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information
java.lang.Exception
at hudson.Proc$LocalProc.join(Proc.java:334)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1818)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
I am trying to adapt a shell script set made for running on Debian 7 to work on Ubuntu 16.
I got to change successfully all except a part that executes PosgreSQL database commands.
Former version of script has these lines:
service postgresql restart
psql -q -U postgres -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -q -U postgres -c "CREATE DATABASE db_crm" -o $log &> /dev/null
When I tried to run psql as above on Ubuntu 16, OS didn't recognize command. It is important to say that script is called with sudo.
I got to find a way to run only database script on Ubuntu 16 changing code so:
service postgresql restart
su postgres <<EOF
psql -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -c "CREATE DATABASE db_crm" -o $log &> /dev/null
However, this same script doesn't work when it is called by main script. Following messages are presented:
here-document at line 41 delimited by end-of-file (wanted 'EOF')
syntax error: unexpected end of file
Even replacing EOF to beggining of next line, error continues.
If there is a way to use psql in shell script without to use EOF would be better.
The reason your script is failing, is you forgot the EOF at the end of input.
su postgres <<EOF
psql -c "DROP DATABASE IF EXISTS db_crm" -o $log &> /dev/null
psql -c "CREATE DATABASE db_crm" -o $log &> /dev/null
EOF #<<<--- HERE
An easy way to do this is to put your commands into a temporary file, then re-direct that into psql. Obviously you don't want this to stop for a password prompt, in this case either use a user that doesn't need it, or set $PGPASSWORD - or prompt at the beginning of the script - there's lots of ways around.
#! /usr/bin/env bash
# PGPASSWORD='' #(set this to stop password prompts, but it's insecure)
PSQL="psql -q -U postgres -o $log" #TODO define $log
TMPFILE="/tmp/sql-tmp.`date +%Y%m%d_%H%M%S_%N`.sql"
# TODO - check $TMPFILE does not exist already
echo "DROP DATABASE IF EXISTS db_crm;" > "$TMPFILE"
echo "CREATE DATABASE db_cr;" >> "$TMPFILE"
# ... etc.
# run the command, throw away any stdout, stderr
PSQL < "$TMPFILE" 2>&1 > /dev/null
# exit with the psql error code
err_code=$?
exit $?
I have troubles restarting a dockerized postgres database (I use Core OS). The database is started in a bash script using the command
# boot.sh
sudo -i -u postgres /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
which works. I have another script called by confd which is run when some etcd keys change (this part is ok, the file is correctly called) and must restart postgres (not reload, because some config changes require a restart). Here are the main options I tried, which failed...
# restart.sh
sudo -u postgres /usr/lib/postgresql/9.3/bin/pg_ctl --pgdata=/var/lib/postgresql/9.3/main restart
systematically raises an error:
%FATAL: lock file "postmaster.pid" already exists
%HINT: Is another postmaster (PID 273) running in data directory "/var/lib/postgresql/9.3/main"?
Furthermore,
# restart.sh
rm /var/lib/postgresql/9.3/main/postmaster.pid
sudo -i -u postgres /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
,
rm /var/lib/postgresql/9.3/main/postmaster.pid
/etc/init.d/postgresql start
,
/etc/init.d/postgresql restart
and
exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"
fail with
ERROR exit status 1
Any thought? Thank you in advance!
For me, changing the config and doing
$ docker restart <postgres_container>
on the host works just fine.
I'm using Fabric to initialize a postgres server. I have to add a "sleep 1" at the end of the command or the postgres server processes die without explanation or an entry in the log:
sudo('%(pgbin)s/pg_ctl -D %(pgdata)s -l /tmp/pg.log restart && sleep 1' % env, user='postgres')
That is, I see this output on the terminal:
[dbserv] Executing task 'setup_postgres'
[dbserv] run: /bin/bash -l -c "sudo -u postgres /usr/lib/postgresql/9.1/bin/pg_ctl -D /data/pg -l /tmp/pg.log restart && sleep 1"
[dbserv] out: waiting for server to shut down.... done
[dbserv] out: server stopped
[dbserv] out: server starting
Without the && sleep 1, there's nothing in /tmp/pg.log (though the file is created), and no postgres processes are running. With the sleep, everything works fine.
(And if I execute the same command directly on target machine's command line, it works fine without the sleep.)
Since it's working, it doesn't really matter, but I'm asking anyway: Does someone know what the sleep is allowing to happen and why?
You might try also using the pty option set it to false and see if it's related to how fabric handles pseudo-ttys.