I am trying to start a postgresql docker container which is of version 10.5.
But before that I have used 9.6 version in the same docker-compose.yml file and there is no data populated in the database.
And now after changing the version of postgres container, I'm not able to run the docker-compose up. It is throwing the below error.
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 9.6,
which is not compatible with this version 10.5 (Debian
10.5-2.pgdg90+1)
This is how the docker-compose.yml file looks like.
version: '2'
services:
postgres_service:
container_name: postgresql_container
restart: always
image: postgres:10.5
volumes:
- postgres-data:/var/lib/postgresql/data
- ./postgresql/init:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=password
volumes:
postgres-data:
driver: local
Can someone please let me know where the issue is. Where am I making mistake?
Do I need to delete any volumes before proceeding with the new postgres version?
I also have postgresql installed in my local.
postgres=# select version();
version
-------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 10.10 (Ubuntu 10.10-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit
(1 row)
Will this cause any issue?
The problem caused because the volume which your compose created to store your database still keep old data which initiated by PostgreSQL 9.6. That volume name is postgres-data which created when you use named volume on your docker-compose.yml. So simply to get rid of this, you can use some ways below:
Using docker-compose command:
Run docker-compose down -v, this will stop all your container inside that compose and remove all named volume inside that compose.
You could take a look at docker-compose down command
Using docker volume command:
Run docker volume ls to get list of current volumes on your machine, I think you will see your volume on that list too:
DRIVER VOLUME NAME
local postgres-data
Run docker volume rm postgres-data to remove that volume, if your container still running and you couldn't remove it then you can use -f to force remove it
Hope that helps!
What worked for me was deleting pgdata folder inside the root of my project and running docker-compose build -d. It then showed me
/usr/local/bin/docker-entrypoint.sh: /docker-entrypoint-initdb.d/create-multiple-postgres-databases.sh: /bin/bash: bad interpreter: Permission denied
To fix it, I ran
chmod +x pg-init-scripts/create-multiple-postgresql-databases.sh
Notice that the .sh file name should match the one you have. And finally, docker-compose up -d.
Related
I know lots of questions sound like this, and they all have the same answer: delete your volumes to force it to reinitialize.
The problem is, I'm being careful to delete my volumes, but it's consistently spinning up the container incorrectly every time.
My docker-compose.yml
version: "3.1"
services:
db:
environment:
- POSTGRES_DB=mydb
- POSTGRES_PASSWORD=changeme
- POSTGRES_USER=myuser
image: postgres
My process:
$ docker volume ls
DRIVER VOLUME NAME
$ docker-compose up -v # or docker-compose up --force-recreate
yet it always creates the "postgres" user instead of myuser. The output when it starts up shows that it "will be owned by user 'postgres'" and I can only docker exec as postgres, not my user.
The instructions seem very straightforward. Am I missing something, or is this a bug?
What happens when you use the compose file above?
I can only docker exec as postgres, not myuser
The environment variable POSTGRES_USER controls the database user, not the linux user. Take a look at the chapter Arbitrary --user Notes in the documentation to learn how to change the linux user.
Mac OS 10.13.6
docker 9.03.5, build 633a0ea
I create two docker containers, web and db using docker-compose.yml. It worked for several months. Recently I decided to rebuild containers from scratch, so I actually removed the existing ones and started over:
$ docker-compose -f docker-compose-dev.yml build --no-cache
$ docker-compose -f docker-compose-dev.yml up -d
Duiring the "build" run, building db container yields this:
DETAIL: The data directory was initialised by PostgreSQL version 9.6,
which is not compatible with this version 11.2.
exited with code 1
The db container does not start so I can not check what it's got inside.
My containers
are defined like this:
version: '3'
services:
web:
restart: unless-stopped
container_name: web
build:
context: .
dockerfile: Dockerfile.dev
ports:
- "8000:8000"
environment:
DJANGO_SETTINGS_MODULE: '<my_app>.settings.postgres'
DB_NAME: 'my_db'
DB_USER: 'my_db_user'
DB_PASS: 'my_db_user'
DB_HOST: 'my_db_host'
PRODUCTION: 'false'
DEBUG: 'True'
depends_on:
- db
volumes:
- ./:/usr/src/app/
db:
image: postgres:11.2-alpine
volumes:
- myapp-db-dev:/var/lib/postgresql/data
environment:
- POSTGRES_DB=<my_db>
- POSTGRES_USER=<my_db_user>
- POSTGRES_PASSWORD=<my_db_password>
volumes:
myapp-db-dev:
My local postgresql is 11.3 (which should be irrelevant):
$ psql --version
psql (PostgreSQL) 11.3
and my local postgresql data directory was removed completely
$ rm -rf /usr/local/var/postgres
However, it's up-to-date:
$ brew postgresql-upgrade-database
Error: postgresql data already upgraded!
I read Stack Overflow 17822974 and Stack Overflow 19076980, those advices did not help.
How to fix this data incompatibility? If possible, I would like to avoid downgrading postgres. I don't even get what data it's talking about at that point, all the data is migrated later in a separate step.
It seems like on the first run Postgres 9.6 was specified as an image. So, the container was initialized and the data was put to the myapp-db-dev named volume. Then someone changed the version and you've got the error. The possible solution would be:
Temporary downgrade the version to the Postgres 9.6, e.g. specify postgres:9.6.
Go to the container and dump the data with pg_dump utility.
Change version to 11.2 and specify new volume (it's a good advice to use host volume).
Restore the data.
I've been playing with Docker for the past week and think the container idea is very useful, but despite reading everything I can for the past 3 days I can't get the volume mapping to work
get docker-compose to use my existing volume.
Docker Version: 18.03.1-ce
docker-compose version 1.21.1, build 7641a569
I created a volume using the following via a Dockerfile
# Reference SQL image
FROM microsoft/mssql-server-windows-developer
# Create directory within SQL container for database files mapped to the volume
VOLUME sqldata:c:/MSSQL
and here it shows:
C:\ProgramData\Docker\volumes>docker volume ls
local sqldata
Now I've tried probably 60+ different "solutions" based on StackOverflow and Docker forums, but none of them work. (Note despite the names below with Azure I am simply trying to get this to run locally, Azure is next hurdle)
Docker-compose.yaml:
version: '3.4'
services:
ws:
image: wsManager
container_name: azure-wcf
ports:
- "80"
depends_on:
- db
db:
image: dbimage:latest
container_name: azure-db
volumes:
- \sqldata:/mssql
# - type: volume
# source: sqldata
# target: /mssql
ports:
- "1433"
I've added a volumes section but it does not help,
volumes:
sqldata:
external:
name: sqldata
changed the - \sqldata:/mssql
to every possible slash .. . ~ whatever. Moved the file to yaml file
to C:\ProgramData\Docker\volumes - basically any suggestion that showed in my search results. The dbImage is a SQL Server image that I need to persist the data from but am wondering what the magic is as nothing I've tried works. Any help is GREATLY appreciated.
I'm running on Windows 10 Pro build 1803.
Why does this have to be so hard?
Than you to whomever knows how to make this actually work.
The solution is to reference the true path on Windows using the volumes: option as below:
sqldb:
image: sqlimage
container_name: azure-db
volumes:
- "C:\\ProgramData\\Docker\\volumes\\sqldata:c:\\mssql"
To persist the data I used the following:
environment:
- "sa_password=ddsql2017##"
- "ACCEPT_EULA=Y"
- 'attach_dbs= {"dbName":"MyDb","dbFiles":"C:\\MSSQL\\MyDb.mdf","C:\\MSSQL\\MyDb.ldf"]}]'
Hope this helps someone else as many of the examples I found searching both on SO and elsewhere did not work for me, and in the Docker forums there are a lot of posts saying mounting volumes not work for Windows.
For those who are using Ubunto WSL:
sudo mkdir /c
sudo mount --bind /mnt/c /c
navigate to your project file use new path ( /c/your-project-path and not /mnt/c/your-project-path)
edit your docker-compose.yml and use relative path for volume : ( like ./src instead of c/your-project-path/src)
docker-compose up
I was struggling with a similar problem when trying to mount a volume to a specific path of my Windows machine: basically it didn't work so every time I restarted my Docker instance I lose all my DB data.
I finally found out that it is because Docker for Windows by default cannot interpret Windows path so the flag COMPOSE_CONVERT_WINDOWS_PATHS has to be activated. To do so:
Run the command "set COMPOSE_CONVERT_WINDOWS_PATHS=1"
Restart Docker
Go to Settings > Shared Drives > Reset credentials and then select drive and then apply
From the command line, kill the containers (docker container rm -f )
Re-run the containers
Hope it helps
If your windows account credentials has been changed, you also have to reset credentials for shared drives. (Settings > Shared Drives > Reset credentials)
In my case, the password was changed by my company security policy.
Are you sure you really need to map to a certain host directory? If not, my solution is to create a volume beforehand and use it in docker-compose.yaml. I use the same scripts for both windows and linux. That is the beauty of docker.
Here is what I did to start both postgres and mysql:
create_db.sh (you can run it in git bash or similiar environment in windows):
docker volume create --name postgres-data -d local
docker volume create --name mysql-data -d local
docker-compose up -d
docker-compose.yaml:
version: '3'
services:
postgres:
image: postgres:latest
environment:
POSTGRES_DB: datasource
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
volumes:
- postgres-data:/var/lib/postgresql/data
mysql:
image: mysql:latest
environment:
MYSQL_DATABASE: 'train'
MYSQL_USER: 'mysql'
MYSQL_PASSWORD: 'mysql'
MYSQL_ROOT_PASSWORD: 'mysql'
ports:
- 3306:3306
volumes:
- mysql-data:/var/lib/mysql
volumes:
postgres-data:
external: true
mysql-data:
external: true
By default it looks that after installing Docker on Windows, sharing of drivers is disabled - so you won't be able to use volumes(that are stored on disks)
Enabling such sharing, through: Docker in tray - right click - Settings, helped to me, volumes started working fine.
Docker on Windows is having strange behavior as Windows has limitations with credentials and also with the virtual machine that Docker is using(Hyper-V , VirtualBox - depending on your Docker version and setup).
Basically, you are correct to map a folder in
volumes:
section in your service:
The path is
version: '3.4'
services:
db:
image: dbimage:latest
container_name: azure-db
volumes:
- c:/Temp/sqldata:/mssql
Important is that you do not need to explicitly create volume in volumes section, but the docker-compose up will create it(the same is for docker run).
Strange thing is that it will never show up in
docker volume ls
but it will be usable with the same files inside windows directory and inside container path /mssql
You can test it with:
docker run --rm -v c:/Temp/sqldata:/data alpine ls /data
or
docker run --rm -v c:/Temp:/data alpine ls /data
If it Disappear, probably it lost the credentials and Reset it via Docker->Settings->Shared Drives->Reset credentials.
I hope it was clear and covered all the aspects for you.
Launch Docker from your windows taskbar
Click on Settings icon on top
Click Resources
Click File Sharing
Click on (+) sign and add path of local folder in which you want to map the container volume.
It worked for me.
I am new to the docker ecosystem and I am trying to spin up a simple postgres container along with a volume so it persists its data, by using a yaml composer file. The file is as follows:
# Use postgres/example user/password credentials
version: '3.3'
services:
db:
image: postgres
environment:
POSTGRES_DB: recrow
POSTGRES_USER: recrow
POSTGRES_PASSWORD: recrow_db_1000
PGDATA: /var/lib/pgsql/data/pgdata
volumes:
- ./pgsql/data:/var/lib/pgsql/data/pgdata
However, upon calling docker-compose -f stack.yml up I get the following error:
fixing permissions on existing directory
/var/lib/postgresql/data/pgdata ... initdb: could not change
permissions of directory "/var/lib/postgresql/data/pgdata": Operation
not permitted
/var/lib/pgsql/data/pgdata is supposed to be a directory relative to the container's root, while ./pgsql/data is a path on the host. I am running the container from an ntfs-3g partition mounted on /mnt/storage. What could be the problem? I am also running docker without root permissions, by adding my user to the docker group and this user also has full access to the beforementioned mount point /mnt/storage.
I'm guessing this is going to be an incompatibility with ntfs-3g. The PostgreSQL image contains an entrypoint script that is doing some permission changes on container start: https://github.com/docker-library/postgres/blob/972294a377463156c8d61297320c872fc7d370a9/9.6/docker-entrypoint.sh#L32-L38. I found another relevant question at https://askubuntu.com/questions/11840/how-do-i-use-chmod-on-an-ntfs-or-fat32-partition that talks about being able to set permissions at mount time. But not being able to change via chmod or chown (which is likely the reason for the failure in this case).
Unfortunately, I think the answer here is that you cannot use ntfs-3g safely for backing Docker host volume mounts.
Following off of #liam-mitchell's note above, that is the answer. Use named volumes such like the following:
services:
db:
image: postgres:12-alpine
volumes:
- "postgres:/data/postgres"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- PGDATA=/data/postgres
...
volumes:
postgres:
I work with OpenShift and had the same problem to run this official image from Docker Hub.
In my case, the solution was to use the official postgres image from red hat repository, the image from red hat repository has fixed this problem, this is can be an alternative.
I had the same issue with docker on WSL2. Setting the :Z flag for the mount and not mounting to a Windows file system directory (/mnt/*) but a linux directory (/home/*) worked for me.
my compose:
version: '3.3'
services:
postgres:
container_name: dbs2-postgres
environment:
- POSTGRES_PASSWORD=mysecretpassword
- PGDATA=/var/lib/postgresql/data/pgdata
volumes:
- './data:/var/lib/postgresql/data:Z'
image: postgres
Here is my docker compose file
version "2"
services:
my_postgres:
image: postgres:9.6
volumes:
- /Users/my_user_name/test_docker/my_volume_space:/var/lib/postgresql
ports:
- "5432:5432"
I entered the following command in mac
docker-machine start
docker-machine env
evcal "$(docker-machine env default)"
docker-compose up
psql -h 192.168.99.100 -p 5432 -U postgres
create table test (my_id bigserial primary key);
INSERT INTO test (my_id) values (1);
SELECT * FROM test;
\q
Originally I thought the above commands will cause a .sql file to be created in ./my_volume_space of the host computer. But I don't see any .sql file in ./my_volume_space rather just an empty data directory in ./my_volume_space
Furthermore if I docker-compose down and docker-compose up again I can see my data in the database is now gone.
I suspected that when I created the data when the image is running, the data is not stored back to ./my_volume_space thus when I reboot, there is nothing to mount from the host.
Can someone tell me what I am doing wrong?
Thanks
Path volumes do not work on docker-machine (macOS) with postgres image, source.
The work-around is to use named volumes. Example docker-compose.yaml:
services:
test-postgres-compose :
...
volumes :
- pgdata:/var/lib/postgresql/data
...
volumes :
pgdata :
Docker compose volumes info
I believe the volume path on the container is to /var/lib/postgresql/data not /var/lib/postgresql