My ubuntu 18.04 has installed postgresql locally. I want to mount existing data to docker.
The below is my docker-compose.yml
version: '3'
services:
db:
image: postgres:10-alpine
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- "15432:5432"
volumes:
- /etc/postgresql/10/main:/var/lib/postgresql/data
I did not change default postgres 10 config in Ubuntu. As default, I think the config is located in /etc/postgresql/10/main and data is located in /var/lib/postgresql/10/main.
If I set
volumes:
- /var/lib/postgresql/10/main:/var/lib/postgresql/data
It warns
db_1 | postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
If I change to
volumes:
- /etc/postgresql/10/main:/var/lib/postgresql/data
It warns
db_1 | initdb: directory "/var/lib/postgresql/data" exists but is not empty
db_1 | If you want to create a new database system, either remove or empty
db_1 | the directory "/var/lib/postgresql/data" or run initdb
db_1 | with an argument other than "/var/lib/postgresql/data".
How to work it out without create a new local docker volumn by docker volume create pgdata
you should be able to specify both into the same directory:
volumes:
- "/var/lib/postgresql/10/main:/var/lib/postgresql/data"
- "/etc/postgresql/10/main/postgresql.conf:/var/lib/postgresql/data/postgresql.conf"
Related
I had postgres 11 installed using docker-compose. I wanted to upgrade it to 12 but even though I have removed the container and its volume but the status of the container says "Restarting".
Here is my docker-compose file
version: '3.5'
services:
postgres:
image: postgres:12
environment:
POSTGRES_HOST_AUTH_METHOD: "trust"
ports:
- "5432"
restart: always
volumes:
- /etc/postgresql/12/postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- db_data:/var/lib/postgresql/data
volumes:
db_data:
However it is not working and the logs has the following issue
2020-07-02T12:54:47.012973448Z The files belonging to this database system will be owned by user "postgres".
2020-07-02T12:54:47.013030445Z This user must also own the server process.
2020-07-02T12:54:47.013068962Z
2020-07-02T12:54:47.013222608Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-02T12:54:47.013261425Z The default database encoding has accordingly been set to "UTF8".
2020-07-02T12:54:47.013281815Z The default text search configuration will be set to "english".
2020-07-02T12:54:47.013293326Z
2020-07-02T12:54:47.013303793Z Data page checksums are disabled.
2020-07-02T12:54:47.013313919Z
2020-07-02T12:54:47.013450079Z initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2020-07-02T12:54:47.013487706Z If you want to create a new database system, either remove or empty
2020-07-02T12:54:47.013501126Z the directory "/var/lib/postgresql/data" or run initdb
2020-07-02T12:54:47.013512379Z with an argument other than "/var/lib/postgresql/data".
How could I remove or empty this /var/lib/postgresql/data when the container is constantly restarting?
Thanks in advance
Quoting #yosifkit from this issue
The volume needs to be empty or a valid already initialized postgres
database with the file PG_VERSION in there so the init can be
skipped.
... If there are any files or folders in there like lost+found it
will probably fail to initialize. If there are files that you want to
keep in the volume (or have no control over) you could adjust the
PGDATA environment variable to point to a sub-directory in there
like -e PGDATA=/var/lib/postgresql/data/db-files/.
So I added PGDATA to the environment section of the compose file to solve the issue (notice the some_name at the end):
services:
postgres:
image: postgres:12
environment:
PGDATA: /var/lib/postgresql/data/some_name/
I got this issue because the /var/lib/postgresql/data/postgresql.conf and the /var/lib/postgresql/data overlap in the docker container at /var/lib/postgresql/.
An example of a broken config is:
version: "3.8"
services:
db:
image: "postgres:10"
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/var/lib/postgresql/data/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
To avoid this I tell PostgreSQL to find it's config in /etc/postgresql.conf instead so the overlapping volumes don't occur like this:
version: "3.8"
services:
db:
image: "postgres:10"
command: ["postgres", "-c", "config_file=/etc/postgresql.conf"]
ports:
- "5432:5432"
volumes:
- ./postgresql.conf:/etc/postgresql.conf
- ./pg-data:/var/lib/postgresql/data
This is similar to what pmsoltani suggests, but I move the location of the postgresql.conf file instead of the data directory.
I had the same issue today, I fixed it by removing the content of the volume db_data in your case
docker volume ls
docker volume inspect db_data <-- will show you the mountpoint
I went to the directory (mountpoint) ex: /path/data
cp data data.backup
cd data
rm -R *
And start services:
docker-compose up -d
Another solution is to simply remove the volume attached to the container:
docker-compose down -v
I am starting a postgres container using following docker-compose.yml
version: '3'
services:
db:
image: postgres:latest
container_name: postgres
environment:
POSTGRES_USER: usr
POSTGRES_PASSWORD: pswd
POSTGRES_DB: db
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
- nfs_cur_dir:/var/lib/postgresql/data
volumes:
nfs_cur_dir:
driver: local
driver_opts:
type: nfs
o: "addr=10.15.187.88,rw"
device: ":/u/uname/home/database"
I am getting following error when starting the container
$sudo ./docker-compose up db
Starting postgres ... done
Attaching to postgres
postgres | mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied
postgres exited with code 1
The permissions on database directory are 777
drwxrwxrwx 3 uname grpname 4096 May 5 22:57 database
After the failure I also see pgdata directory created as this -
drwx------ 2 polkitd root 4096 May 5 22:57 pgdata
Note:
The data directory for the postgres is mapped to an NFS location. Hence I have defined a new NFS volume in the docker-compose and mapped that to the postgres container.
I am using PGDATA env variable to define a different location for the data directory.
Other than above two things there is nothing out of ordinary. If I use a local drive location for the data directory this works fine !
You should check the permissions that NFS share exposes.
According to what you said, if you use a local drive it works fine. That's why I think the NFS share's permissions aren't working as you expect.
Maybe you should create the directory before trying to run your application.
I am tring to start a postgreSQL docker container with my mac; i use OSX 10.11.16 El Capitan with Docker Toolbox 19.03.01.
If i run:
docker run --name my_postgres -v my_dbdata:/var/lib/postgresql/data -p 54320:5432 postgres:11
all was done and i get:
my_postgres | 2019-09-17 04:51:48.908 UTC [41] LOG: database system is ready to accept connections
but if i use an .yml file like this one:
docker-compose.yml:
version: "3"
services:
db:
image: "postgres:11"
container_name: "my_postgres"
ports:
- "54320:5432"
volumes:
- my_dbdata:/var/lib/postgresql/data
volumes:
my_dbdata:
and run
docker-compose up
i get instead:
my_postgres | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
my_postgres |
my_postgres | 2019-09-17 04:51:49.009 UTC [41] LOG: received fast shutdown request
my_postgres | 2019-09-17 04:51:49.011 UTC [41] LOG: aborting any active transactions
my_postgres | waiting for server to shut down....2019-09-17 04:51:49.087 UTC [41] LOG: background worker "logical replication launcher" (PID 48) exited with exit code 1
my_postgres | 2019-09-17 04:51:49.091 UTC [43] LOG: shutting down
my_postgres | 2019-09-17 04:51:49.145 UTC [41] LOG: database system is shut down
Why the same think with docker-compose fail?
So many thanks in advance
Try the below one it worked for me
version: '3.1'
services:
db:
image: postgres
restart: always
environment:
POSTGRES_PASSWORD: mypassword
volumes:
- ./postgres-data:/var/lib/postgresql/data
ports:
- 5432:5432
Then use docker-compose up to get the logs after using the previous command use
docker-compose logs -f
If you are trying to access and existing volume on the host machine, you need to specify that the volume was created outside the Compose file with the external keyword like this:
version: "3.7"
services:
db:
image: postgres
volumes:
- data:/var/lib/postgresql/data
volumes:
data:
external: true
I took the example from the Compose file reference https://docs.docker.com/compose/compose-file/.
Also double check the contents of your external volume between runs, to see if it was overriden.
Please also double check your quotes, you don't need to put the image name in quotes, but I don't think that's the issue.
The my_dbdata named volume is not the same for the two cases.
docker run creates a volume named my_dbdata, instead docker-compose creates by default a volume called <dir>_my_dbdata
Run docker volume to list the volumes:
docker volume ls |grep my_dbdata
I suspect the volume created by docker-compose has issues and as a consequence postgres doesn't start correctly. The initialization of the database in the my_postgres container is done only once.
Try to remove the container and the volume created by docker-compose:
docker rm my_postgres
docker volume rm <dir>_my_dbdata
Hope it helps
I'm experiencing issues using the official Postgres docker image, something that never happened on this machine.
I'm using that through docker-compose but when running docker-compose up, the db container does not start, here's the error:
db_1 | 2019-04-19 22:20:27.180 UTC [1] FATAL: lock file "postmaster.pid" is empty
db_1 | 2019-04-19 22:20:27.180 UTC [1] HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
I tried several times to remove the image so it then will be re-pulled, but hasn't solved the problem.
I'm using Docker for Mac 2.0.0.3 (31259)
here's the docker-compose.yml:
version: '2'
services:
web:
build: .
entrypoint: sh -c "python3 /var/www/my_project/manage.py migrate --noinput &&
python3 /var/www/my_project/manage.py runserver 0.0.0.0:8000"
restart: on-failure
ports:
- "9001:8000"
volumes:
- .:/var/www/my_project
depends_on:
- db
env_file:
- ./.env
environment:
- DB_SERVER=db
- DB_NAME=postgres
- DB_USER=postgres
db:
ports:
- "9432:5432"
image: postgres
volumes:
- database:/var/lib/postgresql
volumes:
database:
Any help on this?
You can try to remove the pid file manually from the volume mounted by the container. Something like this:
Search the volume:
docker inspect container-name
Search for the directory where the volume is stored:
...
"Mounts": [
{
"Type": "volume",
"Name": "9009b4e54686dc36f797a15fd8fc09213f3e8949583b823525881c010f2fe347",
"Source": "/var/lib/docker/volumes/9009b4e54686dc36f797a15fd8fc09213f3e8949583b823525881c010f2fe347/_data",
Now delete the pid file:
rm /var/lib/docker/volumes/9009b4e54686dc36f797a15fd8fc09213f3e8949583b823525881c010f2fe347/_data/
After this you might get this error message:
lock file "/var/run/postgresql/.s.PGSQL.5432.lock" is empty
This can be fixed by overwriting that file with a file containing a number:
echo 12345 > bla
docker cp bla container-name:/var/run/postgresql/.s.PGSQL.5432.lock
docker restart container-name
It could happen because you are starting the PostgreSQL server when you are still running your PostgreSQL server.
Try
pg_ctl stop
then
pg_ctl start
or
pg_ctl restart
I'm trying to create a database and connect to it within my container network. I don't want to have to ssh into a box to create users/databases etc, as this is not a scalable or easily distributable process.
This is what I have so far:
# docker-compose.yml
db:
image: postgres:9.4
volumes:
- ./db/init.sql:/docker-entrypoint-initdb/10-init.sql
environment:
- PGDATA=/tmp
- PGDATABASE=web
- PGUSER=docker
- PGPASSWORD=password
This is my init.sql file:
CREATE DATABASE web;
CREATE USER docker WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE web TO docker;
When I start up the container and try to connect to it, I get this error:
db_1 | FATAL: role "docker" does not exist
db_1 | done
db_1 | server started
db_1 | FATAL: database "web" does not exist
db_1 | psql: FATAL: database "web" does not exist
The first time this happened, I tried to create a role like this:
CREATE ROLE docker with SUPERUSER PASSWORD password;
GRANT web TO docker;
But it did not have any effect. To make matters even more confusing, when I use node-postgres to connect to the db, I get this error:
Error: connect ECONNREFUSED
But how can the connection be refused if the db service isnt even up??
In a nutshell, these are the questions I'm trying to solve:
How can I create a database using only the files in my project (i.e. no manual commands)?
How do I create a user/role using only the files in my project?
How do I connect to this database?
Thank you in advance.
How can I create a database using only the files in my project (i.e.
no manual commands)?
The minimal docker-compose.yml config for you defined user and database is:
postgres:
image: postgres:9.4
environment:
- POSTGRES_DB=web
- POSTGRES_USER=myuser
How do I create a user/role using only the files in my project?
To execute scripts on database initialization take a look at the official docs for initdb.
To get you started with a quick and dirty solution create a new file e.g. init_conf.sh in the same directory as your docker-compose.yml:
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB" <<-EOSQL
CREATE ROLE docker with SUPERUSER PASSWORD 'password';
EOSQL
And add the volumes directive to your docker-compose.yml.
volumes:
- .:/docker-entrypoint-initdb.d
Recreate your container because otherwise, you wouldn't trigger a new database initialization. That means, docker stop and docker rm the old one first before executing docker-compose up again. STDOUT gives you now some information about our newly introduced script.
How do I connect to this database?
To connect to your database with docker exec via the terminal:
docker exec -ti folder_postgres_1 psql -U myuser -d web
A docker-compose.yml in one of my production environments looks like the following:
services:
postgres:
logging: &logging
driver: json-file
options:
max-size: "10m"
max-file: "5"
build: ./docker/postgres # path to custom Dockerfile
volumes:
- postgres_data:/var/lib/postgresql/data
- postgres_backup:/backups
env_file: .env
restart: always
# ... other services like web, celery, redis, etc.
Dockerfile:
FROM postgres:latest
# ...
COPY *.sh /docker-entrypoint-initdb.d/
# ...
The environment variable you are using are wrong. Try this
version: '3.3'
services:
db:
image: postgres:9.4
restart: always
environment:
- POSTGRES_USER=docker
- POSTGRES_PASSWORD=password
- POSTGRES_DB=web
volumes:
- db_data:/var/lib/postgresql/data
# optional port
ports: ["5555:5432"]
volumes:
db_data:
then from any other docker-compose service you can access the DB at db:5432 and from your host machine you can access postgres on localhost:5555 if you also add the ports