I try to run a Postgres Docker container in an Azure Web App.
When i try to mount a volume to the Data folder, i get the error: FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
my Compose script:
version: "3"
services:
db:
image: postgres:11.2
environment:
- POSTGRES_USER=test
- POSTGRES_PASSWORD=test
- POSTGRES_DB=test
volumes:
- ${WEBAPP_STORAGE_HOME}/data:/var/lib/postgresql/data
ports:
- "5433:5432"
Docker host is set to linux.
how can i get around this issue?
(if i dont, the data is lost every restart / container update)
For this issue, you can take a look at the Dockerfile of the image postgres. There is a step that changes the ownership of the directory /var/lib/postgresql with the command:
chown -R postgres:postgres /var/lib/postgresql
But when you use the persistent storage in Azure Web App, you cannot change the permission of it. So it causes the error:
FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
See more details about the limitation here.
Related
I have a Postgres database running on my host. The datafiles for the database is stored at /usr/local/var/postgresql#13.
To get the full system running easily I'd like to have a Docker with a Postgres service running for other Docker apps to connect to. I would however like to have the Docker Postgres service to use the existing datafiles on the host ...
How do I set up the volume correctly to point to the hosts database files?
Do have to have a user/password when running the Docker against existing datafiles?
I have the following but can get the volume to work ...
version: "3.9"
services:
web:
build: .
ports:
- 8081:3011
depends_on:
- db
environment:
- PGHOST=db
- PGDATABASE=loggingtestdb
- PGUSER=postgres
- PGPASSWORD=postgres
db:
image: postgres
ports:
- 5432:5432
volumes:
- /usr/local/var/postgresql#13 <--- Need help here.
How do I map the container pg datafile location to the hosts pg datafile location? 🙏
Update 1
This is the datafile folder for the db on the host
After comments I updated the volumes to below
volumes:
- /usr/local/var/postgresql#13:/var/lib/postgresql/data
But when running docker compose I only get
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /usr/local/var/postgresql#13
Update 2
/use/local works fine. But as soon as I add the /var folder to the path Docker for some reason can’t find it … What am I missing here?
I had postgres 11 installed using docker-compose. I wanted to upgrade it to 12 but even though I have removed the container and its volume but the status of the container says "Restarting".
Here is my docker-compose file
version: '3.5'
services:
postgres:
image: postgres:12
environment:
POSTGRES_HOST_AUTH_METHOD: "trust"
ports:
- "5432"
restart: always
volumes:
- /etc/postgresql/12/postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- db_data:/var/lib/postgresql/data
volumes:
db_data:
However it is not working and the logs has the following issue
2020-07-02T12:54:47.012973448Z The files belonging to this database system will be owned by user "postgres".
2020-07-02T12:54:47.013030445Z This user must also own the server process.
2020-07-02T12:54:47.013068962Z
2020-07-02T12:54:47.013222608Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-02T12:54:47.013261425Z The default database encoding has accordingly been set to "UTF8".
2020-07-02T12:54:47.013281815Z The default text search configuration will be set to "english".
2020-07-02T12:54:47.013293326Z
2020-07-02T12:54:47.013303793Z Data page checksums are disabled.
2020-07-02T12:54:47.013313919Z
2020-07-02T12:54:47.013450079Z initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2020-07-02T12:54:47.013487706Z If you want to create a new database system, either remove or empty
2020-07-02T12:54:47.013501126Z the directory "/var/lib/postgresql/data" or run initdb
2020-07-02T12:54:47.013512379Z with an argument other than "/var/lib/postgresql/data".
How could I remove or empty this /var/lib/postgresql/data when the container is constantly restarting?
Thanks in advance
Quoting #yosifkit from this issue
The volume needs to be empty or a valid already initialized postgres
database with the file PG_VERSION in there so the init can be
skipped.
... If there are any files or folders in there like lost+found it
will probably fail to initialize. If there are files that you want to
keep in the volume (or have no control over) you could adjust the
PGDATA environment variable to point to a sub-directory in there
like -e PGDATA=/var/lib/postgresql/data/db-files/.
So I added PGDATA to the environment section of the compose file to solve the issue (notice the some_name at the end):
services:
postgres:
image: postgres:12
environment:
PGDATA: /var/lib/postgresql/data/some_name/
I got this issue because the /var/lib/postgresql/data/postgresql.conf and the /var/lib/postgresql/data overlap in the docker container at /var/lib/postgresql/.
An example of a broken config is:
version: "3.8"
services:
db:
image: "postgres:10"
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
To avoid this I tell PostgreSQL to find it's config in /etc/postgresql.conf instead so the overlapping volumes don't occur like this:
version: "3.8"
services:
db:
image: "postgres:10"
command: ["postgres", "-c", "config_file=/etc/postgresql.conf"]
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/etc/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
This is similar to what pmsoltani suggests, but I move the location of the postgresql.conf file instead of the data directory.
I had the same issue today, I fixed it by removing the content of the volume db_data in your case
docker volume ls
docker volume inspect db_data <-- will show you the mountpoint
I went to the directory (mountpoint) ex: /path/data
cp data data.backup
cd data
rm -R *
And start services:
docker-compose up -d
Another solution is to simply remove the volume attached to the container:
docker-compose down -v
I have a project where I use google compute engine to host my app and docker to containerize it.
I have a postgres image and I want to use a volume to make my data persistent when I restart the container. Moreover I want the db data to be stored in google storage. So I have a google storage bucket and I have mounted a directory in my google compute engine to that bucked. Specifically what I did is mkdir /home/vetter_leo/data where data is the folder I want to use as a volume and then I mount it using gcsfuse --dir-mode 777 --file-mode 777 -o allow_other --implicit-dirs artifacts.helenos-273112.appspot.com /home/vetter_leo/data/.
My dockerfile for the postgres image is this :
FROM postgres:latest
USER postgres
ENV POSTGRES_USER helenos
ENV POSTGRES_PASSWORD helenos
ENV POSTGRES_DB helenos
ENV PGDATA /var/lib/postgresql/data/pgdata
COPY init_helenos_schema.sql /docker-entrypoint-initdb.d/
EXPOSE 5432
and my docker-compose file is this :
version: "3.5"
services:
postgres:
container_name: postgres
image: postgres
build:
context: .
dockerfile: ./postgres.prod.dockerfile
volumes:
- /home/vetter_leo/data:/var/lib/postgresql/data
networks:
default:
external:
name: helenos-network
When doing docker-compose -f docker-compose.yml up -d --build I end up with the container not being started and this error is shown chmod: changing permissions of '/var/lib/postgresql/data/pgdata': Operation not permitted.
I have searched the web but so far I have not been able to find a solution for my problem. Any help would be greatly appreciated. Thanks.
I ended up using a persistend disk as sugested by #mebius99 and it works so no problem anymore.
I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.
I run:
docker-compose build
then
docker-compose up
And I receive the following error:
ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""
My full docker-compose.yml looks like this (note the service called db where the volume is defined):
version: '3'
services:
nginx:
image: nginx:latest
ports:
- 80:8000
volumes:
- ./src:/src
- ./config/nginx:/etc/nginx/conf.d
- ./src/static:/static
depends_on:
- web
web:
build: .
command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
depends_on:
- db
volumes:
- ./src:/src
- ./src/static:/static
expose:
- 8000
db:
image: postgres:latest
volumes:
- /private/var/lib/postgresql:/var/lib/postgresql
ports:
- 5432:5432
Any ideas for how to solve?
The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:
VOLUME /var/lib/postgresql/data
Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.
For clarity, here is the volume the docker engine created for you:
/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data
And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:
mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied
Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.
As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.
So I'm having a problem mounting an existing set of data for Docker Postgres that I cannot figure out for the life of me. Here's my docker compose file.
version: '2'
services:
postgresql:
image: postgres:9.5
environment:
- PGDATA=/data
ports:
- '5432:5432'
volumes:
- ~/.postgresql:/data
web:
build: .
command: sbt/sbt run
volumes:
- .:/app
ports:
- '9001:9001'
depends_on:
- postgresql
Here's the error I see
ostgresql_1 | FATAL: data directory "/data" has wrong ownership
postgresql_1 | HINT: The server must be started by the user that owns the data directory.
Does anyone have any clue how to fix it? Thank you
PS I am using Docker Machine through OSX if that makes a difference in this problem.
The error message is pretty clear. I think the container runs postgres with user postgres which has a uid/gid of 999 (see https://github.com/docker-library/postgres/blob/3f8e9784438c8fe54f831c301a45f4d55f6fa453/9.5/Dockerfile line 5). You need to chown your host data folder to a user with the same uid.