Docker volume backup and restore postgres:10 image & named volume - postgresql

I'm trying to backup/restore a docker volume. This volume is a named volume attached to a postgres:10 db image. The volume is seeded via a pg_dump/restore. I'm using docker-compose and here is what it looks like:
services:
db:
image: postgres:10
ports:
- '5432:5432'
volumes:
- postgres-data:/var/lib/postgresql/data
volumes:
postgres-data:
driver: local
localstack-data:
driver: local
Here is what the mounted volume looks like on the db service:
"Mounts": [
{
"Type": "volume",
"Name": "postgres-data",
"Source": "/var/lib/docker/volumes/postgres-data/_data",
"Destination": "/var/lib/postgresql/data",
"Driver": "local",
"Mode": "rw",
"RW": true,
"Propagation": ""
}
]
Now, when I say "backup/restore", I mean I want to be able to have a backup of the data inside that volume so that after I have messed all the data up, I can simply just replace it with the backup and have fresh data. A simple "restore w/ snapshot" type of action. I'm following the documentation found on docker.
To test it:
I add a user to my postgres-data
Stop my container docker stop $(docker ps -q)
Perform backup (db_1 is container name): docker run --rm --volumes-from db_1 -v $(pwd):/backup busybox tar cvf /backup/backup.tar /var/lib/postgresql/data
Delete user after backup is complete
Here is an example of the output of backup command that follows w/ NO errors:
tar: removing leading '/' from member names
var/lib/postgresql/data/
var/lib/postgresql/data/PG_VERSION
var/lib/postgresql/data/pg_hba.conf
var/lib/postgresql/data/lib/
var/lib/postgresql/data/lib/postgresql/
var/lib/postgresql/data/lib/postgresql/data/
var/lib/postgresql/data/lib/postgresql/data/PG_VERSION
var/lib/postgresql/data/lib/postgresql/data/pg_hba.conf
var/lib/postgresql/data/lib/postgresql/data/lib/
var/lib/postgresql/data/lib/postgresql/data/lib/postgresql/
var/lib/postgresql/data/lib/postgresql/data/lib/postgresql/data/
var/lib/postgresql/data/lib/postgresql/data/lib/postgresql/data/PG_VERSION
var/lib/postgresql/data/lib/postgresql/data/lib/postgresql/data/pg_hba.conf
var/lib/postgresql/data/lib/postgresql/data/lib/postgresql/data/pg_stat/
Once done, tar file gets created. Now, maybe the error is with the path? If you look at the output I'm seeing lib/postgresql/data nested over and over. Not sure if that is an okay result or not.
Once done, I perform the "restore": docker run --rm --volumes-from db_1 -v $(pwd):/backup busybox sh -c "cd /var/lib/postgresql/data && tar xvf /backup/backup.tar --strip 1"
What I expect: To see the deleted user back in the db. Basically, to see the same data that was in the volume when I performed the backup.
What I'm seeing: the data is not getting restored. Still same data. No user restored and any data that has been changed stays changed.
Again, perhaps what I'm doing is the wrong thing or "overkill" for a simple desire to hot swap out used volumes with a fresh, untouched one. Any ideas would be helpful from debugging or a better approach.

Related

Postgres running in docker-compose unable to write data to mounted volume

Description
I am running a Postgres container in docker-compose. I am mounting the /data directory needed by Postgres into the container using a volume in the docker-compose.yml below.
Qualifications
The Postgres user must be called graph-node and create a database called graph-node
I delete the data/postgres/ folder before each docker-compose up using the boot.sh script below for application-specific reasons. Just know that /data/postgres is re-created on each run of docker-compose up.
Expected Behavior
Postgres boots and writes all files it needs to the mounted /data/postgres volume.
Actual Behavior
Postgres boots fine, but writes nothing to the volume.
Possible Reasons
This feels like a read/write permissions problem? I've added :rw in the third column of the volume as suggested, still no cigar. I run chmod -R a+rwx ./data on the data dir to get access to all files recursively .
The oddest thing to is that if I manually run chmod -R a+rwx ./data after booting, Postgres suddenly IS able to write to the directory all needed files. But if I run this before it's created as seen below (recursively for all things in /data) it does not work.
Files
boot.sh
# Check for data/ dir. If found, make it recursively rwx for all users. Otherwise, create it and make it recursively rwx for all users.
if [ -d "./data" ]
then
chmod -R a+rwx ./data
else
mkdir data
chmod -R a+rwx ./data
fi
if [ -d "./data/postgres" ]
then
rm -rf data/postgres
else
echo "No data/postgres dir found. Proceeding"
fi
docker-compose -f docker-compose.yml up
docker-compose.yml
version: "3"
services:
postgres:
image: postgres
ports:
- '5432:5432'
command:
[
"postgres",
"-cshared_preload_libraries=pg_stat_statements"
]
environment:
POSTGRES_USER: graph-node
POSTGRES_PASSWORD: let-me-in
POSTGRES_DB: graph-node
volumes:
- ./data/postgres:/var/lib/postgresql/data:rw
Machine + Software Specs
Operating System: Windows 10, WSL2, Ubuntu
Docker Version: 20.10.7 (running directly on the machine since it's Ubuntu, NOT in Docker Desktop like on a Mac)
Well, not exactly an answer, but because I only needed one-run ephemeral storage for Postgres (I was deleting the data/ dir between runs anyways) I solved the problem by just removing the external volume and letting Postgres write data into the container itself, where it surely had privileges.

How to wait for Postgres to be ready before loading data in Docker Compose

I have a simple Docker Compose file:
version: "3"
volumes:
pg-data:
services:
db:
image: postgis/postgis
build: ./postgis
volumes:
- pg-data:/var/lib/postgresql
ports:
- 5432:5432
restart: on-failure
My Dockerfile:
FROM postgis/postgis
USER root
COPY docker-entrypoint.sh /docker-entrypoint.sh
COPY data_load.sh /data_load.sh
RUN chmod +x /*.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
My docker-entrypoint.sh:
exec /data_load.sh &
exec /docker-entrypoint.sh
My data_load.sh:
until pg_isready -h localhost -p 5432
do
printf 'PostgreSQL not ready! \n'
sleep 15
done
echo "Loading data"
python /load_data.py &&
echo "Data load complete!"
When running docker-compose up --build I get:
...
Successfully built 88b291d6b47e
Successfully tagged postgis/postgis:latest
Creating my_compose_db_1 ... done
Attaching to my_compose_db_1
db_1 | Starting script to add data
db_1 | localhost:5432 - no response
db_1 | PostgreSQL not ready yet, trying again in 15 seconds
The problem is Postgres never starts! The data_load.sh script executes as I can see the 'PostgreSQL is not ready!' printed out every 15 seconds. Ideas on what I'm doing wrong?
In the script setup you have, nothing ever runs the base image's entrypoint. You provide your own script as /docker-entrypoint.sh, which concludes by re-running itself. (The postgres base image puts its entrypoint script in /usr/local/bin/docker-entrypoint.sh.)
For a PostgreSQL derived image, a much simpler setup will work. At first initialization, the standard postgres image will run any *.sql or *.sh scripts in /docker-entrypoint-initdb.d, and you're guaranteed that the database is running at that particular point. So if you reduce the loader script to just
#!/bin/sh
exec /load_data.py
and copy that into the right directory
FROM postgis/postgis
COPY data_load.sh /docker-entrypoint-initdb.d/data_load.sh
COPY load_data.py /load_data.py
RUN chmod +x /docker-entrypoint-initdb.d/data_load.sh /load_data.py
# And use the base image's ENTRYPOINT/CMD
you don't need to write this scripting by hand at all.
If the script is a more involved database-seeding job, it's also reasonable to run it from a totally separate Docker container, or using of your application's database migration system. (Since schemas do change occasionally, it's probably better to limit the database-local setup to creating initial users and databases and leaving the rest to your application code.)

How can I keep changes I made to Postgresql Docker container?

I'm using the official postgresql docker image to start a container.
Afterwards, I install some software and use psql to create some tables etc. I am doing this by first starting the postgres container as follows:
docker run -it --name="pgtestcontainer" -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:9.6.6
Then I attach to this container with
docker exec -it pgtestcontainer bash
and I install software, create db tables etc.
Afterwards, I first quit from the second terminal session (that I used to install software) and do a ctrl + c in the first one to stop the postgres container.
At this point my expectation is that if I commit this postgres image with
docker commit xyz...zxy pg-commit-test
and then run a new container based on the committed image with:
docker run -it --name="modifiedcontainer" -e POSTGRES_PASSWORD=postgres -p 5432:5432 pg-commit-test
then I should have all the software and tables in place.
The outcome of the process above is that the software I've installed is in the modifiedcontainer but the sql tables etc are gone. So my guess is my approach is more or less correct but there is something specific to postgres docker image I'm missing.
I know that it creates the db from scratch if no external directory or docker volume is bound to
/var/lib/postgresql/data
but I'm not doing that and after the commit I'd expect the contents of the db to stay as they are.
How do I follow the procedure above (or the right one) and keep the changes to database(s)?
The postgres Dockerfile creates a mount point at /var/lib/postgresql/data which you must mount an external volume onto if you want persistent data.
ENV PGDATA /var/lib/postgresql/data
RUN mkdir -p "$PGDATA" && chown -R postgres:postgres "$PGDATA" && chmod 777 "$PGDATA" # this 777 will be replaced by 700 at runtime (allows semi-arbitrary "--user" values)
VOLUME /var/lib/postgresql/data
https://docs.docker.com/engine/reference/builder/#notes-about-specifying-volumes
You can create a volume using
docker volume create mydb
Then you can use it in your container
docker run -it --name="pgtestcontainer" -v mydb:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:9.6.6
https://docs.docker.com/engine/admin/volumes/volumes/#create-and-manage-volumes
In my opinion, the best way is to create your own image with a /docker-entrypoint-initdb.d folder and your script inside.
Look How to extend this image
But without volume you can't (I think) save your datas.
I solved this by passing PGDATA parameter with a value that is different than the path that is bound to docker volume as suggested in one of the responses to this question.

How to persist data in a dockerized postgres database using volumes

My docker compose file has three containers, web, nginx, and postgres. Postgres looks like this:
postgres:
container_name: postgres
restart: always
image: postgres:latest
volumes:
- ./database:/var/lib/postgresql
ports:
- 5432:5432
My goal is to mount a volume which corresponds to a local folder called ./database inside the postgres container as /var/lib/postgres. When I start these containers and insert data into postgres, I verify that /var/lib/postgres/data/base/ is full of the data I'm adding (in the postgres container), but in my local system, ./database only gets a data folder in it, i.e. ./database/data is created, but it's empty. Why?
Notes:
This suggests my above file should work.
This person is using docker services which is interesting
UPDATE 1
Per Nick's suggestion, I did a docker inspect and found:
"Mounts": [
{
"Source": "/Users/alex/Documents/MyApp/database",
"Destination": "/var/lib/postgresql",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Name": "e5bf22471215db058127109053e72e0a423d97b05a2afb4824b411322efd2c35",
"Source": "/var/lib/docker/volumes/e5bf22471215db058127109053e72e0a423d97b05a2afb4824b411322efd2c35/_data",
"Destination": "/var/lib/postgresql/data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
Which makes it seem like the data is being stolen by another volume I didn't code myself. Not sure why that is. Is the postgres image creating that volume for me? If so, is there some way to use that volume instead of the volume I'm mounting when I restart? Otherwise, is there a good way of disabling that other volume and using my own, ./database?
Strangely enough, the solution ended up being to change
volumes:
- ./postgres-data:/var/lib/postgresql
to
volumes:
- ./postgres-data:/var/lib/postgresql/data
You can create a common volume for all Postgres data
docker volume create pgdata
or you can set it to the compose file
version: "3"
services:
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgress
- POSTGRES_DB=postgres
ports:
- "5433:5432"
volumes:
- pgdata:/var/lib/postgresql/data
networks:
- suruse
volumes:
pgdata:
It will create volume name pgdata and mount this volume to container's path.
You can inspect this volume
docker volume inspect pgdata
// output will be
[
{
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/pgdata/_data",
"Name": "pgdata",
"Options": {},
"Scope": "local"
}
]
I would avoid using a relative path. Remember that docker is a daemon/client relationship.
When you are executing the compose, it's essentially just breaking down into various docker client commands, which are then passed to the daemon. That ./database is then relative to the daemon, not the client.
Now, the docker dev team has some back and forth on this issue, but the bottom line is it can have some unexpected results.
In short, don't use a relative path, use an absolute path.
I think you just need to create your volume outside docker first with a docker create -v /location --name and then reuse it.
And by the time I used to use docker a lot, it wasn't possible to use a static docker volume with dockerfile definition so my suggestion is to try the command line (eventually with a script ) .

How to set docker mongo data volume

I want to use Dockerizing MongoDB and store data in local volume.
But .. failed ...
It has mongo:latest images
kerydeMacBook-Pro:~ hu$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mongo latest b11eedbc330f 2 weeks ago 317.4 MB
ubuntu latest 6cc0fc2a5ee3 3 weeks ago 187.9 MB
I want to store the mono data in ~/data. so ---
kerydeMacBook-Pro:~ hu$ docker run -p 27017:27017 -v ~/data:/data/db --name mongo -d mongo
f570073fa3104a54a54f39dbbd900a7c9f74938e2e0f3f731ec8a3140a418c43
But ... it not work...
docker ps -- no daemon mongo
kerydeMacBook-Pro:~ hu$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
try to run "mongo" --failed
kerydeMacBook-Pro:~ hu$ docker exec -it f57 bash
Error response from daemon: Container f57 is not running
docker inspect mongo
kerydeMacBook-Pro:~ hu$ docker inspect mongo
[
{
"Id": "f570073fa3104a54a54f39dbbd900a7c9f74938e2e0f3f731ec8a3140a418c43",
"Created": "2016-02-15T02:19:01.617824401Z",
"Path": "/entrypoint.sh",
"Args": [
"mongod"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 100,
"Error": "",
"StartedAt": "2016-02-15T02:19:01.74102535Z",
"FinishedAt": "2016-02-15T02:19:01.806376434Z"
},
"Mounts": [
{
"Source": "/Users/hushuming/data",
"Destination": "/data/db",
"Mode": "",
"RW": true
},
{
"Name": "365e687c4e42a510878179962bea3c7699b020c575812c6af5a1718eeaf7b57a",
"Source": "/mnt/sda1/var/lib/docker/volumes/365e687c4e42a510878179962bea3c7699b020c575812c6af5a1718eeaf7b57a/_data",
"Destination": "/data/configdb",
"Driver": "local",
"Mode": "",
"RW": true
}
],
If I do not set data volume, mongo image can work!
But, when setting data volume, it can't. Who can help me?
Try and check docker logs to see what was going on when the container stopped and go in "Existed" mode.
See also if specifying the full path for the volume would help:
docker run -p 27017:27017 -v /home/<user>/data:/data/db ...
The OP adds:
docker logs mongo
exception in initAndListen: 98
Unable to create/open lock file: /data/db/mongod.lock
errno:13 Permission denied
Is a mongod instance already running?
terminating 2016-02-15T06:19:17.638+0000
I CONTROL [initandlisten] dbexit: rc: 100
An errno:13 is what issue 30 is about.
This comment adds:
It's a file ownership/permission issue (not related to this docker image), either using boot2docker with VB or a vagrant box with VB.
Nevertheless, I managed to hack the ownership, remounting the /Users shared volume inside boot2docker to uid 999 and gid 999 (which are what mongo docker image uses) and got it to start:
$ boot2docker ssh
$ sudo umount /Users
$ sudo mount -t vboxsf -o uid=999,gid=999 Users /Users
But... mongod crashes due to filesystem type not being supported (mmap not working on vboxsf)
So the actual solution would be to try a DVC: Data Volume Container, because right now the mongodb doc mentions:
MongoDB requires a filesystem that supports fsync() on directories.
For example, HGFS and Virtual Box’s shared folders do not support this
operation.
So:
the mounting to OSX will not work for MongoDB because of the way that virtualbox shared folders work.
For a DVC (Data Volume Container), try docker volume create:
docker volume create mongodbdata
Then use it as:
docker run -p 27017:27017 -v mongodbdata:/data/db ...
And see if that works better.
As I mention in the comments:
A docker volume inspect mongodbdata (see docker volume inspect) will give you its path (that you can then backup if you need)
Per Docker Docs:
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers.
docker volume create mongodbdata
docker run -p 27017:27017 -v mongodbdata:/data/db mongo
Via Docker Compose:
version: '2'
services:
mongodb:
image: mongo:latest
volumes:
- ./<your-local-path>:/data/db
/data/db is the location of the data saved on the container.
<your-local-path> is the location on your machine AKA the host machine where the actual database journal files will be saved.
For anyone running MongoDB container on Windows: as described here, there's an issue when you mount volume from Windows host to MongoDB container using path (we call this local volume).
You can overcome the issue using a Docker volume (volume managed by Docker):
docker volume create mongodata
Or using docker-compose as my preference:
version: "3.4"
services:
....
db:
image: mongo
volumes:
- mongodata:/data/db
restart: unless-stopped
volumes:
mongodata:
Tested on Windows 10 and it works
Found a link: VirtualBox Shared Folders are not supported by mongodb.