I using docker postgres:9.4 image.
I need to know how to backup and restore volume only container.
Created Volume container:
docker run -v /var/lib/postgresql/data --name dbdata postgres:9.4 /bin/true
Using the volume
docker run --name=postgres --volumes-from=dbdata -d -p 6432:5432 postgres:9.4
Backup Volume container
docker run --volumes-from dbdata -v $(pwd):/backup postgres:9.4 tar cvf /backup/backup.tar /var/lib/postgresql/data
Restore volume in new container
docker run --name=dbdata-new --volumes-from dbdata -v $(pwd):/backup ubuntu:14.04 /bin/sh -c 'cd /var/lib/postgresql/data && tar xvf /backup/backup.tar'
Use in the new volume in creating new postgres container:
docker run --name=postgres-new --volumes-from=dbdata-new -d -p 7532:5432 postgres:9.4
Issue: I get the below error in the logs when I run the new container.
initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data"
Not sure what I am doing wrong. Can someone please point out where I am making mistake.
Could not easily reproduce the issue following the steps with very rudiment data (one record one table one new db):
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -c "CREATE DATABASE ttt;"
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -d ttt -c "CREATE table a(b int); insert into a(b) values(1);"
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -d ttt -c "select * from a;"
when I start the postgres-new I get no exceptions in logs and the data seems to be there:
$ psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 7532 -d ttt -c "select * from a;"
b
---
1
(1 row)
data-new --volumes-from dbdata -v $(pwd):/backup ubuntu:14.04 /bin/sh -c 'cd /var/lib/postgresql/data && tar xvf /backup/backup.tar'
The bold text was the problem.
By untaring the backup in the folder /var/lib/postgresql/data ,the tar command was creating /var/lib/postgresql/data inside the above folder.
Thank you for all the help MyKola.
Related
I'm having trouble in creating a Postgres DB using this bash script:
#! /bin/bash
docker pull postgres
docker run --name coverage-postgres -e POSTGRES_PASSWORD=password -p 5432:5432 -d postgres
export CONTAINER_ID=$(sudo docker ps -a | grep coverage-postgres | head -c12)
sleep 2s
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "create user coverage_user with password 'password';"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "create database coverage owner coverage_user;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "grant all privileges on database coverage to coverage_user;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "\c coverage coverage_user" # it seems useless...
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "CREATE TABLE IF NOT EXISTS postal_codes (id,...;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "CREATE UNIQUE INDEX ... ;"
# exit from container
exit
# restart container
docker start $CONTAINER_ID
In particular, the database is created, the user is created, the table is created but... it's not in the coverage db but in postgres db.
I've tried to add "CREATE TABLE coverage.postal_codes" but coverage is a db and not a schema and it didn't work.
I've tried to use: psql -U coverage_user but the system tells me that database coverage_user doesn't exist.
So of course I thought "I have to specify the database of course!". Then I've tried to use: psql -U coverage as the name of the database but this time, the system makes fun of me and, changing its mind, tells me that the role coverage doesn't exists.
I tried a workaround: within the command -c "\c coverage coverage_user" I concatenated the other commands this way:
-c "\c coverage coverage_user; CREATE TABLE...; CREATE UNIQUE INDEX...;"
but, of course, neither this worked at all.
I make a premise: I know there are other ways to do this but I would like to understand what I am missing with these specific commands.
Solution
#! /bin/bash
docker pull postgres
docker run --name coverage-postgres -e POSTGRES_PASSWORD=password -p 5432:5432 -d postgres
export CONTAINER_ID=$(sudo docker ps -a | grep coverage-postgres | head -c12)
sleep 2s
docker exec -it $CONTAINER_ID psql -U postgres -c "create user coverage_user with password 'password';"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U postgres -c "create database coverage owner coverage_user;"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U postgres -c "grant all privileges on database coverage to coverage_user;"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U coverage_user -c "CREATE TABLE IF NOT EXISTS postal_codes (id int)" coverage
Explanation
https://www.postgresql.org/docs/9.2/app-psql.html
psql [option...] [dbname [username]]
Just add dbname after options. And change user as -U option. You can pass dbname also as an rgument -d
I'm trying to get Postgresql 10.0 working in a docker container. I have the following Dockerfile:
FROM postgres:10.0-alpine
RUN apk add openrc --no-cache
USER postgres
RUN /etc/init.d/postgresql start
RUN psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user') THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;"
RUN createdb main
EXPOSE 5432
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
CMD ["/usr/lib/postgresql/10.0/bin/postgres", "-D", "/var/lib/postgresql/10.0/main", "-c", "config_file=/etc/postgresql/10.0/main/postgresql.conf"]
I got following error:
/bin/sh: /etc/init.d/postgresql: not found
Seems that /etc/init.d/postgresql is really missing. What am I doing wrong?
I finished with this Dockerfile:
FROM postgres:10.0-alpine
USER postgres
RUN chmod 0700 /var/lib/postgresql/data &&\
initdb /var/lib/postgresql/data &&\
echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf &&\
echo "listen_addresses='*'" >> /var/lib/postgresql/data/postgresql.conf &&\
pg_ctl start &&\
psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'main'" | grep -q 1 || psql -U postgres -c "CREATE DATABASE main" &&\
psql -c "ALTER USER postgres WITH ENCRYPTED PASSWORD 'mysecurepassword';"
EXPOSE 5432
This dockerfile create database 'main' (if not exists), starts postgres and sets default user password
I was able to do it using exec commands, so I can run postgres in a running container, in this case alpine 3.7. Note $CONTAINER_NAME is the container id from docker ps:
# Install postgresql, create user, db & start the daemon (for testing)
sudo docker exec $CONTAINER_NAME sh -c 'apk add postgresql'
sudo docker exec $CONTAINER_NAME sh -c 'addgroup -S postgres && adduser -S postgres -G postgres'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chmod -R 777 /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'initdb /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log'
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'buildpgpass';\""
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"CREATE DATABASE builddb;\""
You can also include this in your Dockerfile like so:
# Postgres
RUN apk add postgresql=11.1-r0
RUN (addgroup -S postgres && adduser -S postgres -G postgres || true)
RUN mkdir -p /var/lib/postgresql/data
RUN mkdir -p /run/postgresql/
RUN chown -R postgres:postgres /run/postgresql/
RUN chmod -R 777 /var/lib/postgresql/data
RUN chown -R postgres:postgres /var/lib/postgresql/data
RUN su - postgres -c "initdb /var/lib/postgresql/data"
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
RUN su - postgres -c "pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log && psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';\" && psql --command \"CREATE DATABASE builddb;\""
This seems to work great if you want to test something out on the fly.
There are multiple problems with your dockerfile. I guess you are trying to start the server when building the image to create the main database. This won't work since each command in the dockefile will execute in its own layer, and thus when you reach RUN psql ... the database won't be started since it was started in a different layer. So you need to group the commands in one line.
Second problem is that the file /etc/init.d/postgresql does not exist. The server can be started using the postgres command:
RUN postgres &\
psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user')\
THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;" &\
createdb main
I am beginner both of docker and postgresql.
How do I upgrade docker postgresql 9.5 into 9.6 without losing my
current database?
fyi: im using ubuntu version 14 and docker 17.09
Thanks in advance.
To preserve data across a docker container a volume is required. This volume will mount directly onto the file system of the container and be persevered when the container is killed. It sounds though that the container was created without a volume attached. The best way to get that data is to use copy the data folder for the container and move to the host file system. Then create a docker container with the new image. Copy the data directory to the running container's data directory in this case pgdata:/var/lib/postgresql/data
docker cp [containerID]:/var/lib/postgresql/data /home/user/data/data-dir/
docker stop [containerID]
docker run -it --rm -v pgdata:/var/lib/postgresql/data postgres
docker cp /home/user/data/data-dir [containereID]:/var/lib/postgresql/data
In case that doesn't work i would just dump the current databases, and re-upload them to the new container
You do not store database files to external storage (outside of container).
Then i know only 1 way to store your database:
1) Backup database
2) Shutdown postgres 9.5 container
3) Run new postgres 9.6 container
4) Restore backup
You can use pg_dumpall for backuping full database:
pg_dumpall > backupfile
The resulting dump can be restored with psql:
psql -f backup postgres
I know it's been some time since you asked it, but I hope my solution will help future Googlers :)
I've tried to create a solution that is stateless as possible, to be compatible with CI and upgrade scripts.
The script:
Backs up the whole pg instance using pg_dumpall.
Uses the dump to create the new instance using initdb and psql -f.
The only requirement is a volume with some existing pg_data directory in it.
docker stop lms_db_1
DB_NAME=lms
DB_USERNAME=lmsweb
DB_PASSWORD=123456
CURRENT_DATE=$(date +%d-%m-%Y_%H_%M_%S)
MOUNT_PATH=/pg_data
PG_OLD_DATA=/pg_data/11/data
PG_NEW_DATA=/pg_data/13/data
BACKUP_FILENAME=v11.$CURRENT_DATE.sql
BACKUP_PATH=$MOUNT_PATH/backup/$BACKUP_FILENAME
BACKUP_DIR=$(dirname "$BACKUP_PATH")
VOLUME_NAME=lms_db-data-volume
# Step 1: Create a backup
docker run --rm -v $VOLUME_NAME:$MOUNT_PATH \
-e PGDATA=$PG_OLD_DATA \
-e POSTGRES_DB="${DB_NAME:-db}" \
-e POSTGRES_USER="${DB_USERNAME:-postgres}" \
-e POSTGRES_PASSWORD="${DB_PASSWORD:-postgres}" \
postgres:11-alpine \
/bin/bash -c "chown -R postgres:postgres $MOUNT_PATH \
&& su - postgres /bin/bash -c \"/usr/local/bin/pg_ctl -D \\\"\$PGDATA\\\" start\" \
&& mkdir -p \"$BACKUP_DIR\" \
&& pg_dumpall -U $DB_USERNAME -f \"$BACKUP_PATH\" \
&& chown postgres:postgres \"$BACKUP_PATH\""
# Step 2: Create a new database from the backup
docker run --rm -v $VOLUME_NAME:$MOUNT_PATH \
-e PGDATA=$PG_NEW_DATA \
-e POSTGRES_DB="${DB_NAME:-db}" \
-e POSTGRES_USER="${DB_USERNAME:-postgres}" \
-e POSTGRES_PASSWORD="${DB_PASSWORD:-postgres}" \
postgres:13-alpine \
/bin/bash -c "ls -la \"$BACKUP_DIR\" \
&& mkdir -p \"\$PGDATA\" \
&& chown -R postgres:postgres \"\$PGDATA\" \
&& rm -rf $PG_NEW_DATA/* \
&& su - postgres -c \"initdb -D \\\"\$PGDATA\\\"\" \
&& su - postgres -c \"pg_ctl -D \\\"\$PGDATA\\\" -l logfile start\" \
&& su - postgres -c \"psql -f $BACKUP_PATH\" \
&& printf \"\\\nhost all all all md5\\\n\" >> \"\$PGDATA/pg_hba.conf\" \
"
I create backups like this: docker exec DOCKER pg_dump -U USER -F t DB | gzip > ./FILE.tar.gz
What's the best way to restore the database given that the database runs within a container?
For your case:
docker exec -it <CONTAINER> gunzip < backup.tar.gz | pg_restore -U <USER> -F t -d <DB>
Remote restore is also available if your container is public facing and remote connections are allowed in pg_hba.conf for postresql:
gunzip < backup.tar.gz | pg_restore -U <USER> -F t -d <DB> -h <HOST_IP> -p 5432
As a rule of thumb, it is good idea to document your backup and restore commands specific to the project.
How take backup of the data which is existing in the running PostgreSQL container
Create some folder in your root
mkdir -p '/myfolder/bdbackup'
download the postgres image which you are using and execute the following command
docker run --name demo1 -e POSTGRES_PASSWORD=passowrd -v /myfolder/bdbackup:/var/lib/postgresql/data -d postgres
docker exec -it demo1 psql -U postgres
Back up will be stored in the following folder /myfolder/bdbackup
you can kill the container and stop the container any time but data will be stored in the host.
and once again re-run the postgres the container with same command
docker run --name demo2 -e POSTGRES_PASSWORD=passowrd -v /myfolder/bdbackup:/var/lib/postgresql/data -d postgres
docker exec -it demo1 psql -U postgres
and execute following query select * from emp;
you can see the data has restored...
I am using the official postgresql docker image (version 9.4). I have extended the Dockerfile, so I can alter the settings in the postgresql.conf etc, using a bash script. It successfully adds and runs the script on entrypoint, for a single sed command. But when I put 2 or more sed commands, I get the following error:
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/config.sh
: No such file or directoryread
/var/lib/postgresql/data/postgresql.conf
I am trying on Windows 10, in combination with Vagrant and VirtualBox, using NFS file system on shared folders, via the vagrant-winnfsd plugin.
Why is this happening? How can I alter my bash script in order to work with more configuration settings? Is there a better way?
Dockerfile:
FROM postgres:9.4
RUN echo "Europe/Athens" > /etc/timezone \
&& dpkg-reconfigure -f noninteractive tzdata
RUN localedef -i el_GR -c -f UTF-8 -A /usr/share/locale/locale.alias el_GR.UTF-8
ADD config.sh /docker-entrypoint-initdb.d/
RUN chmod 755 /docker-entrypoint-initdb.d/config.sh
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
config.sh:
#!/bin/bash
sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
database.yml
postgres:
container_name: postgres-9.4
image: ***/postgres-9.4
volumes_from:
- postgres_data
ports:
- 5432:5432
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
- USERMAP_UID=999
- USERMAP_GID=999
postgres_data:
container_name: postgres_data
image: ***/postgres-9.4
volumes:
- ./services/postgres:/etc/postgresql
- ./services/postgres:/var/lib/postgresql
- ./services/postgres/logs:/var/log/postgresql
command: "true"
You might want to try using a RUN statement to execute your bash script or just run sed directly with both commands combined with a semicolon:
RUN sed -i -e 's/^#\(logging_collector = \).*/\1on/; s/^\(max_connections = \).*/\11000/' \
/var/lib/postgresql/data/postgresql.conf
A more scalable solution would be to put the sed program in an external file, then use these statements:
ADD postgres-edit.sed /var/local
RUN sed -i -f /var/local/postgres-edit.sed /var/lib/postgresql/data/postgresql.conf
postgres-edit.sed:
# sed script to edit postgresql configuration
s/^#\(logging_collector = \).*/\1on/
s/^\(max_connections = \).*/\11000/
Seems like a duplicate of How to customize the configuration file of the official PostgreSQL docker image?.
Copy-paste of my answer at https://stackoverflow.com/a/40598124/385548.
Inject custom postgresql.conf into postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize PostgreSQL configuration in Docker consistently, I suggest using config_file postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the postgres server with psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;