I had postgres 11 installed using docker-compose. I wanted to upgrade it to 12 but even though I have removed the container and its volume but the status of the container says "Restarting".
Here is my docker-compose file
version: '3.5'
services:
postgres:
image: postgres:12
environment:
POSTGRES_HOST_AUTH_METHOD: "trust"
ports:
- "5432"
restart: always
volumes:
- /etc/postgresql/12/postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- db_data:/var/lib/postgresql/data
volumes:
db_data:
However it is not working and the logs has the following issue
2020-07-02T12:54:47.012973448Z The files belonging to this database system will be owned by user "postgres".
2020-07-02T12:54:47.013030445Z This user must also own the server process.
2020-07-02T12:54:47.013068962Z
2020-07-02T12:54:47.013222608Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-02T12:54:47.013261425Z The default database encoding has accordingly been set to "UTF8".
2020-07-02T12:54:47.013281815Z The default text search configuration will be set to "english".
2020-07-02T12:54:47.013293326Z
2020-07-02T12:54:47.013303793Z Data page checksums are disabled.
2020-07-02T12:54:47.013313919Z
2020-07-02T12:54:47.013450079Z initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2020-07-02T12:54:47.013487706Z If you want to create a new database system, either remove or empty
2020-07-02T12:54:47.013501126Z the directory "/var/lib/postgresql/data" or run initdb
2020-07-02T12:54:47.013512379Z with an argument other than "/var/lib/postgresql/data".
How could I remove or empty this /var/lib/postgresql/data when the container is constantly restarting?
Thanks in advance
Quoting #yosifkit from this issue
The volume needs to be empty or a valid already initialized postgres
database with the file PG_VERSION in there so the init can be
skipped.
... If there are any files or folders in there like lost+found it
will probably fail to initialize. If there are files that you want to
keep in the volume (or have no control over) you could adjust the
PGDATA environment variable to point to a sub-directory in there
like -e PGDATA=/var/lib/postgresql/data/db-files/.
So I added PGDATA to the environment section of the compose file to solve the issue (notice the some_name at the end):
services:
postgres:
image: postgres:12
environment:
PGDATA: /var/lib/postgresql/data/some_name/
I got this issue because the /var/lib/postgresql/data/postgresql.conf and the /var/lib/postgresql/data overlap in the docker container at /var/lib/postgresql/.
An example of a broken config is:
version: "3.8"
services:
db:
image: "postgres:10"
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
To avoid this I tell PostgreSQL to find it's config in /etc/postgresql.conf instead so the overlapping volumes don't occur like this:
version: "3.8"
services:
db:
image: "postgres:10"
command: ["postgres", "-c", "config_file=/etc/postgresql.conf"]
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/etc/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
This is similar to what pmsoltani suggests, but I move the location of the postgresql.conf file instead of the data directory.
I had the same issue today, I fixed it by removing the content of the volume db_data in your case
docker volume ls
docker volume inspect db_data <-- will show you the mountpoint
I went to the directory (mountpoint) ex: /path/data
cp data data.backup
cd data
rm -R *
And start services:
docker-compose up -d
Another solution is to simply remove the volume attached to the container:
docker-compose down -v
Related
I am trying to insert data into postgres using docker.
I have a folder in my code named data which has insert commands and has one file named init.sql.
I want to insert the data from init.sql present in folder data to tables present in docker.
version: '3.1'
services:
postgres:
image: postgres:11.6-alpine
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_PORT: 5432
volumes:
- ./tables:/docker-entrypoint-initdb.d/
- ./data:/var/lib/postgresql/data
volumes:
data: {}
I am trying this but I get the error:
initdb: directory "/var/lib/postgresql/data" exists but is not empty
I think I am not using the correct use case, I am new to docker compose.
But is there any way, my use case can get satisfied?
This is caused by an improper usage of the volumes syntax for your named volume.
In order to mount a named volume you have to just use its name like this:
volumes:
- data:/var/lib/postgresql/data
If your syntax begins with a . then it will be a bind mount from your host.
volumes:
- ./data:/var/lib/postgresql/data
The above code is mounting the host folder data relative to where you your docker-compose.yml is located.
This docker-compose.yml should do what you expect.
version: '3.1'
services:
postgres:
image: postgres:11.6-alpine
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_PORT: 5432
volumes:
- ./tables:/docker-entrypoint-initdb.d/
- data:/var/lib/postgresql/data
volumes:
data:
If for some reason your volume has been created already, with an empty or no database, your first step should be running:
docker-compose down --volumes
From the documentation:
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
From: https://docs.docker.com/compose/reference/down/
I'm trying to run a docker image with PostgreSQL that has a volume configured for persisting data.
docker-compose.yml
version: '3.1'
services:
db:
image: postgres
restart: always
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: example
When I start the container I see the output
fixing permissions on existing directory /var/lib/postgresql/data ... ok
and the data folder is no longer readable for me.
If I elevate myself and access the data directory I can see that the files are there. Furthermore, the command ls -ld data gives me
drwx------ 19 systemd-coredump root 4096 May 17 16:22 data
I can manually set the directory permission with sudo chmod 755 data, but that only works until I restart the container.
Why does this happen, and how can I fix it?
The other answer indeed points to the root cause of the problem, however the help page it points to does not contain a solution. Here is what I came up with to make this work for me:
start the container using your normal docker-compose file, this creates the directory with the hardcoded uid:gid (999:999)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_PASSWORD
stop the container and manually change the ownership to uid:gid you want (I'll use 1000:1000 for this example
$ docker stop postgres
$ sudo chown -R 1000:1000 ./data
Edit your docker file to add your desired uid:gid and start it up again using docker-compose (notice the user:)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
user: 1000:1000
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_password
The reason you can't just use user: from the start is that if the image runs as a different user it fails to create the data files.
On the image documentation page, it does mention a solution to add a volume to expose the /etc/passwd file as read-only in the image when providing --user option, however, that did not work for me with the latest image, as I was getting the following error. In fact none of the three proposed solutions worked for me.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
This is because of what is written in the dockerfile of the postgres image.
From line 15 to 18, you'll see that the group 999 and the user 999 are used. I'm guessing that in your host, they map respectively to systemd-coredump and root.
You need to know that whenever you use a user/group in an image, if the uid/gid exist in your host, then it will be mapped to it.
You can read the documentation on the docker hub from the postgres image here. There is a section Arbitrary --user Notes that explain how it works in the context of this image.
An easier and permanent solution would be as follows:
Add these lines to ~/.bashrc:
export UID=$(id -u)
export GID=$(id -g)
Reload your shell:
$ source ~/.bashrc
Modify your docker-compose.yml as follows:
version: "3.7"
services:
db:
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
user: "${UID}:${GID}"
...
Source
here's what i did:
services:
postgres:
image: postgres:15.1
restart: always
environment:
- POSTGRES_USER=my_user
- POSTGRES_PASSWORD=my_user
- POSTGRES_DB=my_user
user: root
ports:
- "5432:5432"
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
postgres_setup:
image: postgres:15.1
user: root
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
entrypoint: [ "bash", "-c", "chmod 750 -R /var/lib/postgresql/data && chmod 750 -R /etc/postgresql"]
depends_on:
- postgres
pgadmin4:
image: dpage/pgadmin4
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=my_user#admin.com
- PGADMIN_DEFAULT_PASSWORD=my_user
- PGADMIN_LISTEN_ADDRESS=0.0.0.0
user: root
ports:
- "5050:80"
volumes:
- /home/my_user/volumes/pgadmin/data:/var/lib/pgadmin
depends_on:
- postgres_setup
the postgres_setup container just changes permissions and then shuts down
I have been struggling with a similar issue and the answer hit me when trying to work around postgres (static uid per container, configured or 70 by default on alpine, 999 on standard image), and docker limitations (no uid translation of volumes).
The answer is to utilize Linux ACL without any changes to docker-compose.yml user - just keep the default internal container user id.
mkdir -p ./data
sudo setfacl -m u:$(id -u):rwx -R ./data/
docker-compose up -d
or
docker-compose up -d
sudo setfacl -m u:$(id -u):rwx -R ./data/
The order of creating data volume's directory does not matter and as long as ACL is set after it was created, you as a user will be able to access it recursively. You can of course add additional permissions.
To check who has access to data folder simply run:
getfacl ./data
I create an Dockerfile with Postgresql with this code:
FROM postgres:9.4
MAINTAINER Fabio Ebner
ENV POSTGRES_PASSWORD="dna44100"
ENV POSTGRES_PORT=5432
EXPOSE ${POSTGRES_PORT}
COPY init.sql /docker-entrypoint-initdb.d/
so How can I specify to always save my db data in my user Machine? cause with this code everty time I stop the container my data are lost
You will need to mount a volume. pointing your host machine to the container's directory /var/lib/postgresql
Source: docker mounting volumes on host
You need to mount a volume to the data directory of PostgreSQL.
You can use the following, using the docker-compose file:
version: "3"
services:
test-postgresql:
image: postgres:9.4
container_name: test-postgresql
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: dna44100
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
- ./folder-on-host:/var/lib/postgresql/data
With the docker-compose file you can start the container with docker-compose up and stop the container with docker-compose down. The database and settings are saved on the specified directory (./folder-on-host).
If you want to remove the volume you can use the command: docker-compose down -v
You can also use the docker run to mount a volume, using the -v or -volume option:
docker run -v ./folder-on-host:/var/lib/postgresql/data yourimagename
New to Docker and I'm trying to set Postgres and pgadmin4 to run as a single service on docker for Mac inside a virtual machine. Everything works but as soon as I stop the service my data is gone. I'm using a named volume to persist data but probably doing something wrong. What is it?
Here's my setup:
# create my VM
docker-machine create dbvm
# set the right environment
eval $(docker-machine env dbvm)
Here's my docker-compose.yaml file:
version: '3'
services:
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=my_db
volumes:
- pgdata:/pgdata
ports:
- 5432:5432
pgadmin:
image: fenglc/pgadmin4
ports:
- 5050:5050
volumes:
- pgadmindata:/pgadmindata
volumes:
pgdata:
pgadmindata:
With docker-compose.yaml, I run:
docker stack deploy -c docker-compose.yaml dbstack
I can do everything on this setup, but if I run docker stack rm dbstack the data is gone after this, but the volumes still exist.
$ docker volume ls
DRIVER VOLUME NAME
local 0c15b0b22c6b850e8768c14045da166253424dda4df8d2e13df75fd54d833412
local 22bab81d9d1de0e07de97363596b096f944752eba617ff574a0ab525239227f5
local 6da6e29fb98ad0f66d7da6a75dc76066ce014b26ea43567c55ed318fda707105
local dbstack_pgadmindata
local dbstack_pgdata
What am I missing?
Unless you have it in some config not shown, I believe you need to map to the default data location inside the container e.g., pgdata:/var/lib/postgresql/data
#Idg is partially correct. postgres data lives at /var/lib/postgresql/data per the Docker Hub readme.
But for it to work in your named volume, you can't use a path on the left side, so correct value would be:
volumes:
- pgdata:/var/lib/postgresql/data
Then the postgres data will stay in that named volume, on the node it was created on.
So I'm having a problem mounting an existing set of data for Docker Postgres that I cannot figure out for the life of me. Here's my docker compose file.
version: '2'
services:
postgresql:
image: postgres:9.5
environment:
- PGDATA=/data
ports:
- '5432:5432'
volumes:
- ~/.postgresql:/data
web:
build: .
command: sbt/sbt run
volumes:
- .:/app
ports:
- '9001:9001'
depends_on:
- postgresql
Here's the error I see
ostgresql_1 | FATAL: data directory "/data" has wrong ownership
postgresql_1 | HINT: The server must be started by the user that owns the data directory.
Does anyone have any clue how to fix it? Thank you
PS I am using Docker Machine through OSX if that makes a difference in this problem.
The error message is pretty clear. I think the container runs postgres with user postgres which has a uid/gid of 999 (see https://github.com/docker-library/postgres/blob/3f8e9784438c8fe54f831c301a45f4d55f6fa453/9.5/Dockerfile line 5). You need to chown your host data folder to a user with the same uid.