I am starting a postgres container using following docker-compose.yml
version: '3'
services:
db:
image: postgres:latest
container_name: postgres
environment:
POSTGRES_USER: usr
POSTGRES_PASSWORD: pswd
POSTGRES_DB: db
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
- nfs_cur_dir:/var/lib/postgresql/data
volumes:
nfs_cur_dir:
driver: local
driver_opts:
type: nfs
o: "addr=10.15.187.88,rw"
device: ":/u/uname/home/database"
I am getting following error when starting the container
$sudo ./docker-compose up db
Starting postgres ... done
Attaching to postgres
postgres | mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied
postgres exited with code 1
The permissions on database directory are 777
drwxrwxrwx 3 uname grpname 4096 May 5 22:57 database
After the failure I also see pgdata directory created as this -
drwx------ 2 polkitd root 4096 May 5 22:57 pgdata
Note:
The data directory for the postgres is mapped to an NFS location. Hence I have defined a new NFS volume in the docker-compose and mapped that to the postgres container.
I am using PGDATA env variable to define a different location for the data directory.
Other than above two things there is nothing out of ordinary. If I use a local drive location for the data directory this works fine !
You should check the permissions that NFS share exposes.
According to what you said, if you use a local drive it works fine. That's why I think the NFS share's permissions aren't working as you expect.
Maybe you should create the directory before trying to run your application.
Related
I have a Postgres database running on my host. The datafiles for the database is stored at /usr/local/var/postgresql#13.
To get the full system running easily I'd like to have a Docker with a Postgres service running for other Docker apps to connect to. I would however like to have the Docker Postgres service to use the existing datafiles on the host ...
How do I set up the volume correctly to point to the hosts database files?
Do have to have a user/password when running the Docker against existing datafiles?
I have the following but can get the volume to work ...
version: "3.9"
services:
web:
build: .
ports:
- 8081:3011
depends_on:
- db
environment:
- PGHOST=db
- PGDATABASE=loggingtestdb
- PGUSER=postgres
- PGPASSWORD=postgres
db:
image: postgres
ports:
- 5432:5432
volumes:
- /usr/local/var/postgresql#13 <--- Need help here.
How do I map the container pg datafile location to the hosts pg datafile location? 🙏
Update 1
This is the datafile folder for the db on the host
After comments I updated the volumes to below
volumes:
- /usr/local/var/postgresql#13:/var/lib/postgresql/data
But when running docker compose I only get
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /usr/local/var/postgresql#13
Update 2
/use/local works fine. But as soon as I add the /var folder to the path Docker for some reason can’t find it … What am I missing here?
I had postgres 11 installed using docker-compose. I wanted to upgrade it to 12 but even though I have removed the container and its volume but the status of the container says "Restarting".
Here is my docker-compose file
version: '3.5'
services:
postgres:
image: postgres:12
environment:
POSTGRES_HOST_AUTH_METHOD: "trust"
ports:
- "5432"
restart: always
volumes:
- /etc/postgresql/12/postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- db_data:/var/lib/postgresql/data
volumes:
db_data:
However it is not working and the logs has the following issue
2020-07-02T12:54:47.012973448Z The files belonging to this database system will be owned by user "postgres".
2020-07-02T12:54:47.013030445Z This user must also own the server process.
2020-07-02T12:54:47.013068962Z
2020-07-02T12:54:47.013222608Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-02T12:54:47.013261425Z The default database encoding has accordingly been set to "UTF8".
2020-07-02T12:54:47.013281815Z The default text search configuration will be set to "english".
2020-07-02T12:54:47.013293326Z
2020-07-02T12:54:47.013303793Z Data page checksums are disabled.
2020-07-02T12:54:47.013313919Z
2020-07-02T12:54:47.013450079Z initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2020-07-02T12:54:47.013487706Z If you want to create a new database system, either remove or empty
2020-07-02T12:54:47.013501126Z the directory "/var/lib/postgresql/data" or run initdb
2020-07-02T12:54:47.013512379Z with an argument other than "/var/lib/postgresql/data".
How could I remove or empty this /var/lib/postgresql/data when the container is constantly restarting?
Thanks in advance
Quoting #yosifkit from this issue
The volume needs to be empty or a valid already initialized postgres
database with the file PG_VERSION in there so the init can be
skipped.
... If there are any files or folders in there like lost+found it
will probably fail to initialize. If there are files that you want to
keep in the volume (or have no control over) you could adjust the
PGDATA environment variable to point to a sub-directory in there
like -e PGDATA=/var/lib/postgresql/data/db-files/.
So I added PGDATA to the environment section of the compose file to solve the issue (notice the some_name at the end):
services:
postgres:
image: postgres:12
environment:
PGDATA: /var/lib/postgresql/data/some_name/
I got this issue because the /var/lib/postgresql/data/postgresql.conf and the /var/lib/postgresql/data overlap in the docker container at /var/lib/postgresql/.
An example of a broken config is:
version: "3.8"
services:
db:
image: "postgres:10"
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
To avoid this I tell PostgreSQL to find it's config in /etc/postgresql.conf instead so the overlapping volumes don't occur like this:
version: "3.8"
services:
db:
image: "postgres:10"
command: ["postgres", "-c", "config_file=/etc/postgresql.conf"]
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/etc/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
This is similar to what pmsoltani suggests, but I move the location of the postgresql.conf file instead of the data directory.
I had the same issue today, I fixed it by removing the content of the volume db_data in your case
docker volume ls
docker volume inspect db_data <-- will show you the mountpoint
I went to the directory (mountpoint) ex: /path/data
cp data data.backup
cd data
rm -R *
And start services:
docker-compose up -d
Another solution is to simply remove the volume attached to the container:
docker-compose down -v
I have a project where I use google compute engine to host my app and docker to containerize it.
I have a postgres image and I want to use a volume to make my data persistent when I restart the container. Moreover I want the db data to be stored in google storage. So I have a google storage bucket and I have mounted a directory in my google compute engine to that bucked. Specifically what I did is mkdir /home/vetter_leo/data where data is the folder I want to use as a volume and then I mount it using gcsfuse --dir-mode 777 --file-mode 777 -o allow_other --implicit-dirs artifacts.helenos-273112.appspot.com /home/vetter_leo/data/.
My dockerfile for the postgres image is this :
FROM postgres:latest
USER postgres
ENV POSTGRES_USER helenos
ENV POSTGRES_PASSWORD helenos
ENV POSTGRES_DB helenos
ENV PGDATA /var/lib/postgresql/data/pgdata
COPY init_helenos_schema.sql /docker-entrypoint-initdb.d/
EXPOSE 5432
and my docker-compose file is this :
version: "3.5"
services:
postgres:
container_name: postgres
image: postgres
build:
context: .
dockerfile: ./postgres.prod.dockerfile
volumes:
- /home/vetter_leo/data:/var/lib/postgresql/data
networks:
default:
external:
name: helenos-network
When doing docker-compose -f docker-compose.yml up -d --build I end up with the container not being started and this error is shown chmod: changing permissions of '/var/lib/postgresql/data/pgdata': Operation not permitted.
I have searched the web but so far I have not been able to find a solution for my problem. Any help would be greatly appreciated. Thanks.
I ended up using a persistend disk as sugested by #mebius99 and it works so no problem anymore.
My ubuntu 18.04 has installed postgresql locally. I want to mount existing data to docker.
The below is my docker-compose.yml
version: '3'
services:
db:
image: postgres:10-alpine
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- "15432:5432"
volumes:
- /etc/postgresql/10/main:/var/lib/postgresql/data
I did not change default postgres 10 config in Ubuntu. As default, I think the config is located in /etc/postgresql/10/main and data is located in /var/lib/postgresql/10/main.
If I set
volumes:
- /var/lib/postgresql/10/main:/var/lib/postgresql/data
It warns
db_1 | postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
If I change to
volumes:
- /etc/postgresql/10/main:/var/lib/postgresql/data
It warns
db_1 | initdb: directory "/var/lib/postgresql/data" exists but is not empty
db_1 | If you want to create a new database system, either remove or empty
db_1 | the directory "/var/lib/postgresql/data" or run initdb
db_1 | with an argument other than "/var/lib/postgresql/data".
How to work it out without create a new local docker volumn by docker volume create pgdata
you should be able to specify both into the same directory:
volumes:
- "/var/lib/postgresql/10/main:/var/lib/postgresql/data"
- "/etc/postgresql/10/main/postgresql.conf:/var/lib/postgresql/data/postgresql.conf"
I'm trying to run a docker image with PostgreSQL that has a volume configured for persisting data.
docker-compose.yml
version: '3.1'
services:
db:
image: postgres
restart: always
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: example
When I start the container I see the output
fixing permissions on existing directory /var/lib/postgresql/data ... ok
and the data folder is no longer readable for me.
If I elevate myself and access the data directory I can see that the files are there. Furthermore, the command ls -ld data gives me
drwx------ 19 systemd-coredump root 4096 May 17 16:22 data
I can manually set the directory permission with sudo chmod 755 data, but that only works until I restart the container.
Why does this happen, and how can I fix it?
The other answer indeed points to the root cause of the problem, however the help page it points to does not contain a solution. Here is what I came up with to make this work for me:
start the container using your normal docker-compose file, this creates the directory with the hardcoded uid:gid (999:999)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_PASSWORD
stop the container and manually change the ownership to uid:gid you want (I'll use 1000:1000 for this example
$ docker stop postgres
$ sudo chown -R 1000:1000 ./data
Edit your docker file to add your desired uid:gid and start it up again using docker-compose (notice the user:)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
user: 1000:1000
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_password
The reason you can't just use user: from the start is that if the image runs as a different user it fails to create the data files.
On the image documentation page, it does mention a solution to add a volume to expose the /etc/passwd file as read-only in the image when providing --user option, however, that did not work for me with the latest image, as I was getting the following error. In fact none of the three proposed solutions worked for me.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
This is because of what is written in the dockerfile of the postgres image.
From line 15 to 18, you'll see that the group 999 and the user 999 are used. I'm guessing that in your host, they map respectively to systemd-coredump and root.
You need to know that whenever you use a user/group in an image, if the uid/gid exist in your host, then it will be mapped to it.
You can read the documentation on the docker hub from the postgres image here. There is a section Arbitrary --user Notes that explain how it works in the context of this image.
An easier and permanent solution would be as follows:
Add these lines to ~/.bashrc:
export UID=$(id -u)
export GID=$(id -g)
Reload your shell:
$ source ~/.bashrc
Modify your docker-compose.yml as follows:
version: "3.7"
services:
db:
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
user: "${UID}:${GID}"
...
Source
here's what i did:
services:
postgres:
image: postgres:15.1
restart: always
environment:
- POSTGRES_USER=my_user
- POSTGRES_PASSWORD=my_user
- POSTGRES_DB=my_user
user: root
ports:
- "5432:5432"
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
postgres_setup:
image: postgres:15.1
user: root
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
entrypoint: [ "bash", "-c", "chmod 750 -R /var/lib/postgresql/data && chmod 750 -R /etc/postgresql"]
depends_on:
- postgres
pgadmin4:
image: dpage/pgadmin4
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=my_user#admin.com
- PGADMIN_DEFAULT_PASSWORD=my_user
- PGADMIN_LISTEN_ADDRESS=0.0.0.0
user: root
ports:
- "5050:80"
volumes:
- /home/my_user/volumes/pgadmin/data:/var/lib/pgadmin
depends_on:
- postgres_setup
the postgres_setup container just changes permissions and then shuts down
I have been struggling with a similar issue and the answer hit me when trying to work around postgres (static uid per container, configured or 70 by default on alpine, 999 on standard image), and docker limitations (no uid translation of volumes).
The answer is to utilize Linux ACL without any changes to docker-compose.yml user - just keep the default internal container user id.
mkdir -p ./data
sudo setfacl -m u:$(id -u):rwx -R ./data/
docker-compose up -d
or
docker-compose up -d
sudo setfacl -m u:$(id -u):rwx -R ./data/
The order of creating data volume's directory does not matter and as long as ACL is set after it was created, you as a user will be able to access it recursively. You can of course add additional permissions.
To check who has access to data folder simply run:
getfacl ./data