I am wondering how I can check whether the DB already exists before trying to create it in the following Docker built line. At the moment, it does not let the build to proceed because the DB has already been created once.
RUN set -e && \
nohup bash -c "docker-entrypoint.sh postgres &" && \
/tmp/wait-for-pg-is-ready.sh && \
psql -U postgres -c "CREATE USER ${DBUSER} WITH SUPERUSER CREATEDB CREATEROLE ENCRYPTED PASSWORD '${DBUSER_PWD}';" && \
psql -U ${DBUSER} -d ${POSTGRES_DB} -c "CREATE DATABASE ${DBNAME} TEMPLATE template0;" && \
pg_restore -v --no-owner --role=${DBUSER} --exit-on-error -U ${DBUSER} -d ${DBNAME} /tmp/pgdump.pgdump && \
psql -U postgres -c "ALTER USER ${DBUSER} WITH NOSUPERUSER;" && \
rm -rf /tmp/pgdump.pgdump
Related
What code should I use if I want to execute a batch file with the psql copy command, (to transfer data from one postgres database table to another)?
How do I connect to the databases with two different host names at once?
How do I set the connection parameters?
I tried using this:
psql \
-U user_name \
-h production_server \
-d database_name \
-c "\\copy users to stdout" | \
psql -U user_name \
-h staging_server \
-d database_name \
-c "\\copy users from stdin"
I have tried multiple way to append log but not able to get.
for example.
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump >db_bkp_01_10_2019.log
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump 2>db_bkp_01_10_2019.log
pg_restore -U postgres -p5333 -d demodb < db_bkp_01_10_2019.dump 2&>db_bkp_01_10_2019.log
I want log related to restoration like what command executed on db while restoration.
You need to use the --verbose
Try this command
./pg_restore -U postgres -p 5333 -d demodb < db_bkp_01_10_2019.dump --verbose 2>db_bkp_01_10_2019.log
Took me sometime to figure this out, it works for me in psql 11 for log:
$pg_restore -h 127.0.0.1 -U user -p 5432 -d dbname -Fc import_data.sql>import.log 2>&1
I'm trying to get Postgresql 10.0 working in a docker container. I have the following Dockerfile:
FROM postgres:10.0-alpine
RUN apk add openrc --no-cache
USER postgres
RUN /etc/init.d/postgresql start
RUN psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user') THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;"
RUN createdb main
EXPOSE 5432
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
CMD ["/usr/lib/postgresql/10.0/bin/postgres", "-D", "/var/lib/postgresql/10.0/main", "-c", "config_file=/etc/postgresql/10.0/main/postgresql.conf"]
I got following error:
/bin/sh: /etc/init.d/postgresql: not found
Seems that /etc/init.d/postgresql is really missing. What am I doing wrong?
I finished with this Dockerfile:
FROM postgres:10.0-alpine
USER postgres
RUN chmod 0700 /var/lib/postgresql/data &&\
initdb /var/lib/postgresql/data &&\
echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf &&\
echo "listen_addresses='*'" >> /var/lib/postgresql/data/postgresql.conf &&\
pg_ctl start &&\
psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'main'" | grep -q 1 || psql -U postgres -c "CREATE DATABASE main" &&\
psql -c "ALTER USER postgres WITH ENCRYPTED PASSWORD 'mysecurepassword';"
EXPOSE 5432
This dockerfile create database 'main' (if not exists), starts postgres and sets default user password
I was able to do it using exec commands, so I can run postgres in a running container, in this case alpine 3.7. Note $CONTAINER_NAME is the container id from docker ps:
# Install postgresql, create user, db & start the daemon (for testing)
sudo docker exec $CONTAINER_NAME sh -c 'apk add postgresql'
sudo docker exec $CONTAINER_NAME sh -c 'addgroup -S postgres && adduser -S postgres -G postgres'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chmod -R 777 /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'initdb /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log'
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'buildpgpass';\""
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"CREATE DATABASE builddb;\""
You can also include this in your Dockerfile like so:
# Postgres
RUN apk add postgresql=11.1-r0
RUN (addgroup -S postgres && adduser -S postgres -G postgres || true)
RUN mkdir -p /var/lib/postgresql/data
RUN mkdir -p /run/postgresql/
RUN chown -R postgres:postgres /run/postgresql/
RUN chmod -R 777 /var/lib/postgresql/data
RUN chown -R postgres:postgres /var/lib/postgresql/data
RUN su - postgres -c "initdb /var/lib/postgresql/data"
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
RUN su - postgres -c "pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log && psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';\" && psql --command \"CREATE DATABASE builddb;\""
This seems to work great if you want to test something out on the fly.
There are multiple problems with your dockerfile. I guess you are trying to start the server when building the image to create the main database. This won't work since each command in the dockefile will execute in its own layer, and thus when you reach RUN psql ... the database won't be started since it was started in a different layer. So you need to group the commands in one line.
Second problem is that the file /etc/init.d/postgresql does not exist. The server can be started using the postgres command:
RUN postgres &\
psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user')\
THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;" &\
createdb main
I am beginner both of docker and postgresql.
How do I upgrade docker postgresql 9.5 into 9.6 without losing my
current database?
fyi: im using ubuntu version 14 and docker 17.09
Thanks in advance.
To preserve data across a docker container a volume is required. This volume will mount directly onto the file system of the container and be persevered when the container is killed. It sounds though that the container was created without a volume attached. The best way to get that data is to use copy the data folder for the container and move to the host file system. Then create a docker container with the new image. Copy the data directory to the running container's data directory in this case pgdata:/var/lib/postgresql/data
docker cp [containerID]:/var/lib/postgresql/data /home/user/data/data-dir/
docker stop [containerID]
docker run -it --rm -v pgdata:/var/lib/postgresql/data postgres
docker cp /home/user/data/data-dir [containereID]:/var/lib/postgresql/data
In case that doesn't work i would just dump the current databases, and re-upload them to the new container
You do not store database files to external storage (outside of container).
Then i know only 1 way to store your database:
1) Backup database
2) Shutdown postgres 9.5 container
3) Run new postgres 9.6 container
4) Restore backup
You can use pg_dumpall for backuping full database:
pg_dumpall > backupfile
The resulting dump can be restored with psql:
psql -f backup postgres
I know it's been some time since you asked it, but I hope my solution will help future Googlers :)
I've tried to create a solution that is stateless as possible, to be compatible with CI and upgrade scripts.
The script:
Backs up the whole pg instance using pg_dumpall.
Uses the dump to create the new instance using initdb and psql -f.
The only requirement is a volume with some existing pg_data directory in it.
docker stop lms_db_1
DB_NAME=lms
DB_USERNAME=lmsweb
DB_PASSWORD=123456
CURRENT_DATE=$(date +%d-%m-%Y_%H_%M_%S)
MOUNT_PATH=/pg_data
PG_OLD_DATA=/pg_data/11/data
PG_NEW_DATA=/pg_data/13/data
BACKUP_FILENAME=v11.$CURRENT_DATE.sql
BACKUP_PATH=$MOUNT_PATH/backup/$BACKUP_FILENAME
BACKUP_DIR=$(dirname "$BACKUP_PATH")
VOLUME_NAME=lms_db-data-volume
# Step 1: Create a backup
docker run --rm -v $VOLUME_NAME:$MOUNT_PATH \
-e PGDATA=$PG_OLD_DATA \
-e POSTGRES_DB="${DB_NAME:-db}" \
-e POSTGRES_USER="${DB_USERNAME:-postgres}" \
-e POSTGRES_PASSWORD="${DB_PASSWORD:-postgres}" \
postgres:11-alpine \
/bin/bash -c "chown -R postgres:postgres $MOUNT_PATH \
&& su - postgres /bin/bash -c \"/usr/local/bin/pg_ctl -D \\\"\$PGDATA\\\" start\" \
&& mkdir -p \"$BACKUP_DIR\" \
&& pg_dumpall -U $DB_USERNAME -f \"$BACKUP_PATH\" \
&& chown postgres:postgres \"$BACKUP_PATH\""
# Step 2: Create a new database from the backup
docker run --rm -v $VOLUME_NAME:$MOUNT_PATH \
-e PGDATA=$PG_NEW_DATA \
-e POSTGRES_DB="${DB_NAME:-db}" \
-e POSTGRES_USER="${DB_USERNAME:-postgres}" \
-e POSTGRES_PASSWORD="${DB_PASSWORD:-postgres}" \
postgres:13-alpine \
/bin/bash -c "ls -la \"$BACKUP_DIR\" \
&& mkdir -p \"\$PGDATA\" \
&& chown -R postgres:postgres \"\$PGDATA\" \
&& rm -rf $PG_NEW_DATA/* \
&& su - postgres -c \"initdb -D \\\"\$PGDATA\\\"\" \
&& su - postgres -c \"pg_ctl -D \\\"\$PGDATA\\\" -l logfile start\" \
&& su - postgres -c \"psql -f $BACKUP_PATH\" \
&& printf \"\\\nhost all all all md5\\\n\" >> \"\$PGDATA/pg_hba.conf\" \
"
I tried:
ssh root#mytutumnode "docker exec -ti mydockercontainerid su postgres -c 'psql mydatabasename'" < mydump.sql
but it executes only
ssh root#mytutumnode "docker exec -ti mydockercontainerid su postgres -c 'psql mydatabasename'"
... and omits < mydump.sql part.
psql --list and alikes do work.
Here's the Makefile i ended up using:
# Convenience cli shortcuts
LOCAL_DOCKER_IP = 192.168.99.100
STAGING_IP = DOCKER_NODE_IP_HERE
RELEASE_IP = IP_FROM_THE_SERVER_WITH_THE_ORIGINAL_DB
DOCKER_DB_ID = $(shell ssh root#$$STAGING_IP "docker ps | grep database" | awk '{print $$1}')
download_staging_db:
psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres -c 'drop database mydb'
psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres -c 'create database mydb'
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'pg_dump mydb'" | psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres mydb
upload_release_db_to_staging_db:
ssh root#$(RELEASE_IP) "cd / && su postgres -c 'pg_dump my db'" > temp.sql
scp temp.sql root#$(STAGING_IP):/temp.sql
ssh root#$(STAGING_IP) "docker cp /temp.sql $(DOCKER_DB_ID):/temp.sql"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'drop database mydb'"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'create database mydb'"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'psql mydb < /temp.sql'"
rm temp.sql