I am trying to raise a service in my docker-compose to use postgres, I can raise the container correctly, the problem is that it does not allow me to authenticate with the password that I am setting in the environment variables
this is my docker-compose service:
version: "3"
services:
api:
build:
context: .
dockerfile: docker/nodejs/Dockerfile
ports:
- "3000:3000"
volumes:
- .:/app
links:
- postgresql
postgresql:
image: postgres:11-alpine
restart: always
environment:
POSTGRES_PASSWORD: 00000000
volumes:
- ~/docker/postgresql:/var/lib/postgresql/data
ports:
- "5432:5432"
what can be the problem here? i trying to get in with pgadmin. In the console show me the error:
password authentication failed for user "postgres"
Password does not match for user "postgres".
The Postgres password is set on the first initialization if the db (when the var/lib/postgresql/data folder is empty). Since you probably ran docker-compose before with a different password, the new one will not take effect.
To fix this, remove the volume in your host machine and have docker-compose automatically recreate it next time you run docker-compose up. Note that removing this volume will delete all data stored in PostgreSQL
Another way to fail with POSTGRES_PASSWORD is to set a password with forbidden characters. You won't get any warnings, but you won't be able to login with this password.
This might have to do with your volumes. Try pruning your volumes and then bring up your containers again. MAKE SURE YOU DON'T NEED THE VOLUMES BEFORE YOU PRUNE THEM!
To see what existing volumes you have
docker volume ls
And then clear them out if you can
docker volume prune
If you are not runing it first time, you could already change it.
You can manualy edit pg_hba.conf file and set trust authorisation type
Related
I'm using docker for the first time to set up a test database that my team can then use. I'm having some trouble getting my data on DBeaver after running my docker-compose file. The issue I'm facing is that my database does not show up in DBeaver (along with relevant Schemas and Tables that I also create/populate in my initialization sql script).
Here is my docker-compose.yml
version: "3"
services:
test_database:
image: postgres:latest
build:
context: ./
dockerfile: Dockerfile
restart: always
ports:
- 5432:5432
environment:
- POSTGRES_USER=dev
- POSTGRES_PASSWORD=test1234
- POSTGRES_DB=testdb
container_name: test_database
In this, I specify the docker file I want it to use for building. Here is the dockerfile:
# syntax = docker/dockerfile:1.3
FROM postgres:latest
ADD test_data.tar .
COPY init_test_db.sql /docker-entrypoint-initdb.d/
Now, when I run docker-compose build and docker-compose up, I can see through the logs that my SQL commands (CREATE, COPY, etc.) do get executed and the rows do get added. But when I connect to this instance through DBeaver, I can't see this at all. In fact, the only database on there is the default Postgres database, even through the logs say I'm connected to test_database.
I followed some other solutions and used docker volume prune as well, but that didn't affect anything (I read some solutions about clearing up volumes, and at that point, I had volumes: /tmp:/tmp as well). Any ideas?
Wow, this wasn't an error after all. All I had to do was go on the connection settings on DBeaver and check 'Show all databases' under the Postgres tab. Hope this can help someone :)
I am not able to login into my postgres databse deployed in docker. PFB my docker-compose.yml
discountdb:
image: postgres
docker-compose.override.yml
discountdb:
container_name: discountdb
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=admin1234
- POSTGRES_DB=DiscountDb
restart: always
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data/
When I am trying to create server from pgAdmin4, I am getting the following error
Following is the db logs
What did I miss?
The environment variables you set are only used if Postgres can't find an existing database when then container starts.
Since you map a docker volume to /var/lib/postgresql/data/, chances are that you already have a database with existing users defined.
Try removing the volume mapping, so you're sure that Postgres creates a fresh database. If that solves it, then you have two options:
If you don't need the data in the volume, you can delete the postgres_data volume so Postgres creates a new database
If you need the data, you need to find out what userid/password you need to use to access the existing database in the volume
Nothing is wrong here but the POSTGRES_PASSWORD is taken into account the first time you start the container I mean when your postgres_data folder is still empty. If you changed the password but postgres_data is not empty the new password is ignored, you must log in with the first one.
The issue was with my postgres_data volume. I removed the volume using the command
docker volume rm -f discountdb
then ran the docker-compose again which resolved the issue.
I set up a Github codespaces environment using devcontainer.json and docker-compose.yaml. Everything works fine, but the postgres database defined in docker-compose.yml loses its data every time the container needs to be re-built.
Here's the bottom part of the docker-compose.yml
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
as you can see, I am trying to map the postgres data volume into a postgres-data volume, but this doesn't work for some reason.
What am I doing wrong that's preventing postgres data from persisting between container builds?
Another option would be to look into using Spawn. (Disclaimer - I'm one of the devs working on it).
We've written some documentation about exactly how to use Spawn-hosted databases with GitHub codespaces here: https://docs.spawn.cc/blog/2021/08/01/spawn-and-codespaces
This will allow you to provision a database thats independent from the GitHub codespace and preserve data between restarts.
You get some extra features with Spawn like arbitrary save points, resets and loading back to saved revisions with Spawn - but the key functionality of spinning up a database for a GitHub codespace and preserving data is one of the things it works extremely well for.
according to https://docs.github.com/en/codespaces/customizing-your-codespace/configuring-codespaces-for-your-project#dockerfile ,
only docker images can be pulled from source and set-up, nowhere they mention that volume persistence is guaranteed.
and after going through this https://code.visualstudio.com/docs/remote/devcontainerjson-reference looks like mounts and few other features related to volumes are not supported for codespaces.
workspaceMount : Not yet supported in Codespaces or when using Clone Repository in Container Volume.
workaround :
in .devcontainer folder where your dockerfile is present add a line like this
RUN curl https://<your_public_cloud>/your_volume.vol -O
here <your_public_cloud> can be google drive, aws or any endpoint where you have access to download the volume. its also the volume you needed to be persist.
and once its downloaded you can mount the volume to postgres service or make a hotswap.
and when you want to save, just upload the volume to your cloud storage provider.
repeat the process every time you build, and save and upload before "unbuild" or dismissing your codespace whatever you like to call.
hope that eases your issue, happy coding!
As long as you don't remove the volume with docker-compose down --volumes as an example, the data should persist.
I had the same issue; and it turned out that I had put a crontab running docker system prune -af every 15 minutes!
You could just mount a host directory, instead of using a docker volume:
volumes:
- /home/me/postgres_data:/var/lib/postgresql/data
This guarantees that no volume cleanup (accidental or deliberate) nukes your database.
Indeed the postgres docs do this in their examples. See the PGDATA environment variable.
As you don't have access to VM, maybe the directory containing your docker-compose.yml changes.
In that case, volume name may change too.
Indeed, by default, your volume name would be the following :
<directory_name>_postgres-data
Could you try a named volume (starting with compose 3.4):
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
external: false
name: postgres-data
documentation of docker-compose can be found here :
https://docs.docker.com/compose/compose-file/compose-file-v3/#name
EDIT 1
If your VM is created at each build, docker dependencies too.
volumes, networks, etc...
A persistent volume is needed somewhere (surviving VM builds).
You may have to create a directory in your local workspace, like:
/local/workspace/postgres-data/
which become in codespaces according to my understanding :
./postgres-data
Check permissions, your user may not exist in the container.
As a result your compose file become:
db:
image: postgres:latest
restart: unless-stopped
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
I am running a simple django webapp with docker-compose. I define both a web service and a db service in a docker-compose.yml file:
version: "3.8"
services:
db:
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
env_file:
- ./.env.dev
depends_on:
- db
volumes:
postgres_data:
I start the service by running:
docker-compose up -d
I can load some data in there with a custom django command that I wrote for my app. Everything is running fine (with data) on localhost:8000.
However, when I run
docker-compose down
(so without -v) and then again
docker-compose up -d
the database is empty again. The volume was not persisted. From what I read in the docker-compose docs and also in several posts here at SO, persisting the volume and reusing it when you start a new container should be the default behavior (which, if I understand it correctly, you can disable by using the --renew-anon-volumes flag).
However in my case, the volume is not persisted. Or maybe it is, but my data is gone.
By doing docker volume ls I can see that my volume (I'll use the name my_volume here) still exists after the docker-compose down command. However, the CreatedAt value has been changed. This makes me think it's a different volume with the same name, and my data is already gone, but I don't know how to confirm that.
This SO answer suggests to mount the volume on /var/lib/postgresql instead of /var/lib/postgresql/data. However, I've seen other resources (like this one) where the opposite is suggested. I've tried both, but neither option works.
Thanks for any advice.
It turns out that the Dockerfile of my app was using an entrypoint in which the following command was executed: python manage.py flush which clears all data in the database. As this gets executed every time the app container starts, it clears all data. It had nothing to do with docker-compose.
I am new to the docker ecosystem and I am trying to spin up a simple postgres container along with a volume so it persists its data, by using a yaml composer file. The file is as follows:
# Use postgres/example user/password credentials
version: '3.3'
services:
db:
image: postgres
environment:
POSTGRES_DB: recrow
POSTGRES_USER: recrow
POSTGRES_PASSWORD: recrow_db_1000
PGDATA: /var/lib/pgsql/data/pgdata
volumes:
- ./pgsql/data:/var/lib/pgsql/data/pgdata
However, upon calling docker-compose -f stack.yml up I get the following error:
fixing permissions on existing directory
/var/lib/postgresql/data/pgdata ... initdb: could not change
permissions of directory "/var/lib/postgresql/data/pgdata": Operation
not permitted
/var/lib/pgsql/data/pgdata is supposed to be a directory relative to the container's root, while ./pgsql/data is a path on the host. I am running the container from an ntfs-3g partition mounted on /mnt/storage. What could be the problem? I am also running docker without root permissions, by adding my user to the docker group and this user also has full access to the beforementioned mount point /mnt/storage.
I'm guessing this is going to be an incompatibility with ntfs-3g. The PostgreSQL image contains an entrypoint script that is doing some permission changes on container start: https://github.com/docker-library/postgres/blob/972294a377463156c8d61297320c872fc7d370a9/9.6/docker-entrypoint.sh#L32-L38. I found another relevant question at https://askubuntu.com/questions/11840/how-do-i-use-chmod-on-an-ntfs-or-fat32-partition that talks about being able to set permissions at mount time. But not being able to change via chmod or chown (which is likely the reason for the failure in this case).
Unfortunately, I think the answer here is that you cannot use ntfs-3g safely for backing Docker host volume mounts.
Following off of #liam-mitchell's note above, that is the answer. Use named volumes such like the following:
services:
db:
image: postgres:12-alpine
volumes:
- "postgres:/data/postgres"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- PGDATA=/data/postgres
...
volumes:
postgres:
I work with OpenShift and had the same problem to run this official image from Docker Hub.
In my case, the solution was to use the official postgres image from red hat repository, the image from red hat repository has fixed this problem, this is can be an alternative.
I had the same issue with docker on WSL2. Setting the :Z flag for the mount and not mounting to a Windows file system directory (/mnt/*) but a linux directory (/home/*) worked for me.
my compose:
version: '3.3'
services:
postgres:
container_name: dbs2-postgres
environment:
- POSTGRES_PASSWORD=mysecretpassword
- PGDATA=/var/lib/postgresql/data/pgdata
volumes:
- './data:/var/lib/postgresql/data:Z'
image: postgres