Mac OS 10.13.6
docker 9.03.5, build 633a0ea
I create two docker containers, web and db using docker-compose.yml. It worked for several months. Recently I decided to rebuild containers from scratch, so I actually removed the existing ones and started over:
$ docker-compose -f docker-compose-dev.yml build --no-cache
$ docker-compose -f docker-compose-dev.yml up -d
Duiring the "build" run, building db container yields this:
DETAIL: The data directory was initialised by PostgreSQL version 9.6,
which is not compatible with this version 11.2.
exited with code 1
The db container does not start so I can not check what it's got inside.
My containers
are defined like this:
version: '3'
services:
web:
restart: unless-stopped
container_name: web
build:
context: .
dockerfile: Dockerfile.dev
ports:
- "8000:8000"
environment:
DJANGO_SETTINGS_MODULE: '<my_app>.settings.postgres'
DB_NAME: 'my_db'
DB_USER: 'my_db_user'
DB_PASS: 'my_db_user'
DB_HOST: 'my_db_host'
PRODUCTION: 'false'
DEBUG: 'True'
depends_on:
- db
volumes:
- ./:/usr/src/app/
db:
image: postgres:11.2-alpine
volumes:
- myapp-db-dev:/var/lib/postgresql/data
environment:
- POSTGRES_DB=<my_db>
- POSTGRES_USER=<my_db_user>
- POSTGRES_PASSWORD=<my_db_password>
volumes:
myapp-db-dev:
My local postgresql is 11.3 (which should be irrelevant):
$ psql --version
psql (PostgreSQL) 11.3
and my local postgresql data directory was removed completely
$ rm -rf /usr/local/var/postgres
However, it's up-to-date:
$ brew postgresql-upgrade-database
Error: postgresql data already upgraded!
I read Stack Overflow 17822974 and Stack Overflow 19076980, those advices did not help.
How to fix this data incompatibility? If possible, I would like to avoid downgrading postgres. I don't even get what data it's talking about at that point, all the data is migrated later in a separate step.
It seems like on the first run Postgres 9.6 was specified as an image. So, the container was initialized and the data was put to the myapp-db-dev named volume. Then someone changed the version and you've got the error. The possible solution would be:
Temporary downgrade the version to the Postgres 9.6, e.g. specify postgres:9.6.
Go to the container and dump the data with pg_dump utility.
Change version to 11.2 and specify new volume (it's a good advice to use host volume).
Restore the data.
Related
I'm trying to build a docker-compose file that will spin up my EF Core web api project, connecting to my Postgres database.
I'm having a hard time getting the EF project connecting to the database.
This is what I currently have for my docker-compose.yml:
version: '3.8'
services:
web:
container_name: 'mybackendcontainer'
image: 'myuser/mybackend:0.0.6'
build:
context: .
dockerfile: backend.dockerfile
ports:
- 8080:80
depends_on:
- postgres
networks:
- mybackend-network
postgres:
container_name: 'postgres'
image: 'postgres:latest'
environment:
- POSTGRES_USER=username
- POSTGRES_PASSWORD=MySuperSecurePassword!
- POSTGRES_DB=MyDatabase
networks:
- mybackend-network
expose:
- 5432
volumes:
- ./db-data/:/var/lib/postgresql/data/
pgadmin:
image: dpage/pgadmin4
ports:
- 15433:80
env_file:
- .env
depends_on:
- postgres
networks:
- mybackend-network
volumes:
- ./pgadmin-data/:/var/lib/pgadmin/
networks:
mybackend-network:
driver: bridge
And my web project docker file looks like this:
# Get base DSK Image from Microsoft
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app
# Copy the CSPROJ file and restore any dependencies (via NUGET)
COPY *.csproj ./
RUN dotnet restore
# Copy the project files and build our release
COPY . ./
RUN dotnet publish -c Release -o out
# Generate runtime image - do not include the whole SDK to save image space
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
EXPOSE 80
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "MyBackend.dll"]
And my connection string looks like this:
User ID =bootcampdb;Password=MySuperSecurePassword!;Server=postgres;Port=5432;Database=MyDatabase; Integrated Security=true;Pooling=true;
Currently I have two problems:
I'm getting Npgsql.PostgresException (0x80004005): 57P03: the database system is starting up when I do docker-compose -up. I tried to add the healthcheck to my postgress db but that did not work. When I go to my Docker desktop app, and start my backend again, that message goes away and I get my second problem...
Secondly after the DB started it's saying: FATAL: password authentication failed for user "username". It looks like it's not creating my user for the database. I even changed not to use .env files but have the value in my docker-compose file, but its still not working. I've tried to do docker-compose down -v to ensure my volumes gets deleted.
Sorry these might be silly questions, I'm still new to containerization and trying to get this to work.
Any help will be appreciated!
Problem 1: Having depends_on only means that docker-compose will wait until your postgres container is started before it starts the web container. The postgres container needs some time to get ready to accept connections and if you attempt to connect before it's ready, you get the error you're seeing. You need to code your backend in a way that it'll wait until Postgres is ready by retrying the connection with a delay.
Problem 2: Postgres only creates the user and database if no database already exists. You probably have an existing database in ./db-data/ on the host. Try deleting ./db-data/ and Postgres should create the user and database using the environment variables you've set.
Goal:
Run Postgres in docker by pulling postgres from docker hub (https://hub.docker.com/_/postgres)
Background:
I get a message when I tried running docker with postgres
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
I got info at https://andrew.hawker.io/dailies/2020/02/25/postgres-uninitialized-error/ about why.
Problem:
"Update your docker-compose.yml or corresponding configuration with the POSTGRES_HOST_AUTH_METHOD environment variable to revert back to previous behavior or implement a proper password." (https://andrew.hawker.io/dailies/2020/02/25/postgres-uninitialized-error/)
I don't understand the solution about how to solve the current situation.
Where can i find the dokcer-compose.yml?
Info:
*I'm newbie in PostGre and Docker
If you need to run PostgreSQL in docker you will have to use a variable in docker run command like this :
$ docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres
The error is telling you the same.
Read more at https://hub.docker.com/_/postgres
Docker-compose.yml is just another option. You can run it just by docker run like in first answer. If you want use docker-compose, in documentation is example of it:
stack.yaml
version: '3.1'
services:
db:
image: postgres
restart: always
environment:
POSTGRES_PASSWORD: example
adminer:
image: adminer
restart: always
ports:
- 8080:8080
and run: docker-compose -f stack.yml up.
Everything is here:
https://hub.docker.com/_/postgres
I am running a simple django webapp with docker-compose. I define both a web service and a db service in a docker-compose.yml file:
version: "3.8"
services:
db:
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
env_file:
- ./.env.dev
depends_on:
- db
volumes:
postgres_data:
I start the service by running:
docker-compose up -d
I can load some data in there with a custom django command that I wrote for my app. Everything is running fine (with data) on localhost:8000.
However, when I run
docker-compose down
(so without -v) and then again
docker-compose up -d
the database is empty again. The volume was not persisted. From what I read in the docker-compose docs and also in several posts here at SO, persisting the volume and reusing it when you start a new container should be the default behavior (which, if I understand it correctly, you can disable by using the --renew-anon-volumes flag).
However in my case, the volume is not persisted. Or maybe it is, but my data is gone.
By doing docker volume ls I can see that my volume (I'll use the name my_volume here) still exists after the docker-compose down command. However, the CreatedAt value has been changed. This makes me think it's a different volume with the same name, and my data is already gone, but I don't know how to confirm that.
This SO answer suggests to mount the volume on /var/lib/postgresql instead of /var/lib/postgresql/data. However, I've seen other resources (like this one) where the opposite is suggested. I've tried both, but neither option works.
Thanks for any advice.
It turns out that the Dockerfile of my app was using an entrypoint in which the following command was executed: python manage.py flush which clears all data in the database. As this gets executed every time the app container starts, it clears all data. It had nothing to do with docker-compose.
I am trying to start a postgresql docker container which is of version 10.5.
But before that I have used 9.6 version in the same docker-compose.yml file and there is no data populated in the database.
And now after changing the version of postgres container, I'm not able to run the docker-compose up. It is throwing the below error.
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 9.6,
which is not compatible with this version 10.5 (Debian
10.5-2.pgdg90+1)
This is how the docker-compose.yml file looks like.
version: '2'
services:
postgres_service:
container_name: postgresql_container
restart: always
image: postgres:10.5
volumes:
- postgres-data:/var/lib/postgresql/data
- ./postgresql/init:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=password
volumes:
postgres-data:
driver: local
Can someone please let me know where the issue is. Where am I making mistake?
Do I need to delete any volumes before proceeding with the new postgres version?
I also have postgresql installed in my local.
postgres=# select version();
version
-------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 10.10 (Ubuntu 10.10-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit
(1 row)
Will this cause any issue?
The problem caused because the volume which your compose created to store your database still keep old data which initiated by PostgreSQL 9.6. That volume name is postgres-data which created when you use named volume on your docker-compose.yml. So simply to get rid of this, you can use some ways below:
Using docker-compose command:
Run docker-compose down -v, this will stop all your container inside that compose and remove all named volume inside that compose.
You could take a look at docker-compose down command
Using docker volume command:
Run docker volume ls to get list of current volumes on your machine, I think you will see your volume on that list too:
DRIVER VOLUME NAME
local postgres-data
Run docker volume rm postgres-data to remove that volume, if your container still running and you couldn't remove it then you can use -f to force remove it
Hope that helps!
What worked for me was deleting pgdata folder inside the root of my project and running docker-compose build -d. It then showed me
/usr/local/bin/docker-entrypoint.sh: /docker-entrypoint-initdb.d/create-multiple-postgres-databases.sh: /bin/bash: bad interpreter: Permission denied
To fix it, I ran
chmod +x pg-init-scripts/create-multiple-postgresql-databases.sh
Notice that the .sh file name should match the one you have. And finally, docker-compose up -d.
I am new to the docker ecosystem and I am trying to spin up a simple postgres container along with a volume so it persists its data, by using a yaml composer file. The file is as follows:
# Use postgres/example user/password credentials
version: '3.3'
services:
db:
image: postgres
environment:
POSTGRES_DB: recrow
POSTGRES_USER: recrow
POSTGRES_PASSWORD: recrow_db_1000
PGDATA: /var/lib/pgsql/data/pgdata
volumes:
- ./pgsql/data:/var/lib/pgsql/data/pgdata
However, upon calling docker-compose -f stack.yml up I get the following error:
fixing permissions on existing directory
/var/lib/postgresql/data/pgdata ... initdb: could not change
permissions of directory "/var/lib/postgresql/data/pgdata": Operation
not permitted
/var/lib/pgsql/data/pgdata is supposed to be a directory relative to the container's root, while ./pgsql/data is a path on the host. I am running the container from an ntfs-3g partition mounted on /mnt/storage. What could be the problem? I am also running docker without root permissions, by adding my user to the docker group and this user also has full access to the beforementioned mount point /mnt/storage.
I'm guessing this is going to be an incompatibility with ntfs-3g. The PostgreSQL image contains an entrypoint script that is doing some permission changes on container start: https://github.com/docker-library/postgres/blob/972294a377463156c8d61297320c872fc7d370a9/9.6/docker-entrypoint.sh#L32-L38. I found another relevant question at https://askubuntu.com/questions/11840/how-do-i-use-chmod-on-an-ntfs-or-fat32-partition that talks about being able to set permissions at mount time. But not being able to change via chmod or chown (which is likely the reason for the failure in this case).
Unfortunately, I think the answer here is that you cannot use ntfs-3g safely for backing Docker host volume mounts.
Following off of #liam-mitchell's note above, that is the answer. Use named volumes such like the following:
services:
db:
image: postgres:12-alpine
volumes:
- "postgres:/data/postgres"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- PGDATA=/data/postgres
...
volumes:
postgres:
I work with OpenShift and had the same problem to run this official image from Docker Hub.
In my case, the solution was to use the official postgres image from red hat repository, the image from red hat repository has fixed this problem, this is can be an alternative.
I had the same issue with docker on WSL2. Setting the :Z flag for the mount and not mounting to a Windows file system directory (/mnt/*) but a linux directory (/home/*) worked for me.
my compose:
version: '3.3'
services:
postgres:
container_name: dbs2-postgres
environment:
- POSTGRES_PASSWORD=mysecretpassword
- PGDATA=/var/lib/postgresql/data/pgdata
volumes:
- './data:/var/lib/postgresql/data:Z'
image: postgres