Description
I am running a Postgres container in docker-compose. I am mounting the /data directory needed by Postgres into the container using a volume in the docker-compose.yml below.
Qualifications
The Postgres user must be called graph-node and create a database called graph-node
I delete the data/postgres/ folder before each docker-compose up using the boot.sh script below for application-specific reasons. Just know that /data/postgres is re-created on each run of docker-compose up.
Expected Behavior
Postgres boots and writes all files it needs to the mounted /data/postgres volume.
Actual Behavior
Postgres boots fine, but writes nothing to the volume.
Possible Reasons
This feels like a read/write permissions problem? I've added :rw in the third column of the volume as suggested, still no cigar. I run chmod -R a+rwx ./data on the data dir to get access to all files recursively .
The oddest thing to is that if I manually run chmod -R a+rwx ./data after booting, Postgres suddenly IS able to write to the directory all needed files. But if I run this before it's created as seen below (recursively for all things in /data) it does not work.
Files
boot.sh
# Check for data/ dir. If found, make it recursively rwx for all users. Otherwise, create it and make it recursively rwx for all users.
if [ -d "./data" ]
then
chmod -R a+rwx ./data
else
mkdir data
chmod -R a+rwx ./data
fi
if [ -d "./data/postgres" ]
then
rm -rf data/postgres
else
echo "No data/postgres dir found. Proceeding"
fi
docker-compose -f docker-compose.yml up
docker-compose.yml
version: "3"
services:
postgres:
image: postgres
ports:
- '5432:5432'
command:
[
"postgres",
"-cshared_preload_libraries=pg_stat_statements"
]
environment:
POSTGRES_USER: graph-node
POSTGRES_PASSWORD: let-me-in
POSTGRES_DB: graph-node
volumes:
- ./data/postgres:/var/lib/postgresql/data:rw
Machine + Software Specs
Operating System: Windows 10, WSL2, Ubuntu
Docker Version: 20.10.7 (running directly on the machine since it's Ubuntu, NOT in Docker Desktop like on a Mac)
Well, not exactly an answer, but because I only needed one-run ephemeral storage for Postgres (I was deleting the data/ dir between runs anyways) I solved the problem by just removing the external volume and letting Postgres write data into the container itself, where it surely had privileges.
Related
I am beginner docker user.
I installed docker and postgresql in Mac OS.
and why most of documents mention the directory
/var/lib/postgresql/data as an volume setting???
Because in my local directory, there is not existed /var/lib/postgresql..
Is it default option? or am I missing something?
Yes, correct, /var/lib/postgresql does not exist on your local computer, but it does in the created container. The volumes parameter is used to associate the local data with the container data, in order to preserve the data in case the container crashes
For example:
volumes:
- ./../database/main:/var/lib/postgresql/data
Above we link the local directory from the left side to the container directory
If you are using official PostgreSQL image from Docker Hub, then you can check the contents of its Dockerfile. E.g. here is a fragment of postgres:15 image responsible for data directory:
ENV PGDATA /var/lib/postgresql/data
# this 777 will be replaced by 700 at runtime (allows semi-arbitrary "--user" values)
RUN mkdir -p "$PGDATA" && chown -R postgres:postgres "$PGDATA" && chmod 777 "$PGDATA"
VOLUME /var/lib/postgresql/data
As you can see Postgres is configured to have data in that directory. And to persist the data even if container is stopped and removed, the volume is created. Volumes have lifetime independent of the container which allows them to "survive".
I want to create two Docker volumes and have their data be persistent. I run sudo docker compose up -d, post some data to my website (text that is stores in a sqlite database and an image stored in the filesystem), then run sudo docker compose down. When I run sudo docker compose up -d again, all the data I posted is gone. With the following configs, I expect the data to still be present.
Dockerfile:
FROM python:3.9.16-buster
RUN pip install --upgrade pip
# The Debian/Buster default is to disable the password.
RUN adduser nonroot
RUN mkdir /home/site/ && chown -R nonroot:nonroot /home/site
RUN chown -R nonroot:nonroot /var/log/site
# two volumes created
VOLUME /home/site/db /home/site/static
WORKDIR /home/site
USER nonroot
# folders ./site/static and ./site/db exist in my host directory
COPY --chown=nonroot:nonroot . .
CMD ["python", "./site/main.py"]
compose.yaml:
services:
site:
build: flask
restart: always
ports:
- '8081:8081'
volumes:
- site_db:/home/site/db # same path as the volumes created in the Dockerfile
- site_static:/home/site/static
command: gunicorn -w 1 -t 3 -b 0.0.0.0:8081 --chdir ./site main:app
volumes:
site_db: # I find it odd these volumes keys don't have values, but that's what I have see other people do
site_static:
docker compose up and docker compose down delete my volumes.
docker compose start and docker compose stop do NOT delete my volumes.
Through the Flask app, check where you are uploading the files to, as well as where the sqlite3 db file is. If these paths do not align with the volumes paths, data will not persist.
I have a docker-compose.yml which I am using to deploy to a remote host from my local (Mac) machine using docker context. The compose config is as follows:
database:
image: postgres:14.2
restart: on-failure
volumes:
- ./db-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
In order to persist data, I have defined a volume ./db-data:/var/lib/postgresql/data. This db-data folder does not exist in my local machine. I want delete this mount completely because I don't want any of the previously persisted data. I know I can define a new volume directory but I would like to use the same directory name (db-data). I have tried the following:
docker compose down --volume --remove-orphans - when I recreate new container, previously persisted data still exists
There is no folder called ./db-data in my Mac working directory.
I tried searching var/lib/docker in my Mac. But that directory does not exists.
Docker for Mac app doesn't list any volumes
There is no db-data in the remote host where the database is deployed
Running docker inspect <container-id> listed the mount directory for the container. The mount directory resembled absolute path of my local computer. For example it was like /Users/<user-name>/dir/db-data. When I saw this I assumed this had to be in the local computer due to the prefix Users/<user-name> but this path was actually found in the root of the remote machine.
Thats because the directory for docker volumes is in the docker vm for MACOS.
Where is /var/lib/docker on Mac/OS X
You would have to follow this to see the volume
I have a simple Docker Compose file:
version: "3"
volumes:
pg-data:
services:
db:
image: postgis/postgis
build: ./postgis
volumes:
- pg-data:/var/lib/postgresql
ports:
- 5432:5432
restart: on-failure
My Dockerfile:
FROM postgis/postgis
USER root
COPY docker-entrypoint.sh /docker-entrypoint.sh
COPY data_load.sh /data_load.sh
RUN chmod +x /*.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
My docker-entrypoint.sh:
exec /data_load.sh &
exec /docker-entrypoint.sh
My data_load.sh:
until pg_isready -h localhost -p 5432
do
printf 'PostgreSQL not ready! \n'
sleep 15
done
echo "Loading data"
python /load_data.py &&
echo "Data load complete!"
When running docker-compose up --build I get:
...
Successfully built 88b291d6b47e
Successfully tagged postgis/postgis:latest
Creating my_compose_db_1 ... done
Attaching to my_compose_db_1
db_1 | Starting script to add data
db_1 | localhost:5432 - no response
db_1 | PostgreSQL not ready yet, trying again in 15 seconds
The problem is Postgres never starts! The data_load.sh script executes as I can see the 'PostgreSQL is not ready!' printed out every 15 seconds. Ideas on what I'm doing wrong?
In the script setup you have, nothing ever runs the base image's entrypoint. You provide your own script as /docker-entrypoint.sh, which concludes by re-running itself. (The postgres base image puts its entrypoint script in /usr/local/bin/docker-entrypoint.sh.)
For a PostgreSQL derived image, a much simpler setup will work. At first initialization, the standard postgres image will run any *.sql or *.sh scripts in /docker-entrypoint-initdb.d, and you're guaranteed that the database is running at that particular point. So if you reduce the loader script to just
#!/bin/sh
exec /load_data.py
and copy that into the right directory
FROM postgis/postgis
COPY data_load.sh /docker-entrypoint-initdb.d/data_load.sh
COPY load_data.py /load_data.py
RUN chmod +x /docker-entrypoint-initdb.d/data_load.sh /load_data.py
# And use the base image's ENTRYPOINT/CMD
you don't need to write this scripting by hand at all.
If the script is a more involved database-seeding job, it's also reasonable to run it from a totally separate Docker container, or using of your application's database migration system. (Since schemas do change occasionally, it's probably better to limit the database-local setup to creating initial users and databases and leaving the rest to your application code.)
I'm trying to run postgresql in docker container, but of course I need to have my database data to be persistent, so I'm trying to use data only container which expose volume to store database at this place.
So, my data container has such Dockerfile:
FROM ubuntu
# Create data directory
RUN mkdir -p /data/postgresql
# Create /data volume
VOLUME /data/postgresql
Which I run:
docker run --name postgresql_data lyapun/postgresql_data true
In my postgresql.conf I set:
data_directory = '/data/postgresql'
Then I run my postgresql container in such way:
docker run -d --name postgre --volumes-from postgresql_data lyapun/postgresql
And I got:
2014-07-04 07:45:57 GMT FATAL: data directory "/data/postgresql" has wrong ownership
2014-07-04 07:45:57 GMT HINT: The server must be started by the user that owns the data directory.
How to deal with this issue? I googled a lot to find some information about using postgresql with docker volumes, but I didn't found anything.
Thanks!
Ok, seems like I found workaround for this issue.
Instead of running postgres in such way:
CMD ["/usr/lib/postgresql/9.1/bin/postgres", "-D", "/var/lib/postgresql/9.1/main", "-c", "config_file=/etc/postgresql/9.1/main/postgresql.conf"]
I wrote bash script:
chown -Rf postgres:postgres /data/postgresql
chmod -R 700 /data/postgresql
sudo -u postgres /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
And replaced CMD in postgresql image to:
CMD ["bash", "/run.sh"]
It works!
You have to set ownership of directory /data/postgresql to the same user, under which you are running your postgresql binary. For example, in Ubuntu it is usually postgres user.
Then you have to use this command:
chown postgres.postgres /data/postgresql
A better way to solve that issue, assuming your postgres images is named "postgres" and that your backup is ./backup.tar:
First, add this to your postgres Dockerfile:
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
Then run:
docker run -it --name postgres -v $(pwd):/db postgres sh -c "tar xvf /db/backup.tar --no-overwrite-dir" && \
docker run -it --name data --volumes-from postgres busybox true && \
docker rm postgres && \
docker run -it --name postgres --volumes-from=data postgres
You don't have permission issues since the archive is extracted by the postgres user of your postgres image, so it is the owner of the extracted files.
You can then backup your data using the data container. The advantage of this solution is that you don't chmod/chown every time you run the image.
This type of errors is quite common when you link a NTFS directory into your docker container. NTFS directories don't support ext3 file & directory access control.
The only way to make it work is to link directory from a ext3 drive into your container.
I got a bit desperate when I played around Apache / PHP containers with linking the www folder. After I linked files reside on a ext3 filesystem the problem disappear.
I published a short Docker tutorial on youtube, may it helps to understand this problem: https://www.youtube.com/watch?v=eS9O05TTFjM