Trying to set up a Django multicontainer application for local development purposes. I haven't found a way to customize pg_hba.conf and postgresql.conf directly from the compose file. I don't want to rely on external scripts or even external files to be copied to the container since the pg_hba.conf and postgresql.conf are rather simple and additional files would add clutter in the project directory. Having a second Dockerfile for the database also is not an optimal solution for me since, as stated, I just need a few lines to be configured/uncommented on those target files.
Below, the current configuration for my compose file.
version: '3.8'
services:
#-------------------------------------------------------#
# postgres
suapdb:
hostname: 'suapdb'
ports:
- "5432:5432"
environment:
POSTGRES_USER: postgres
POSTGRES_HOST_AUTH_METHOD: trust
POSTGRES_PASSWORD: ''
POSTGRES_DB: suap_dev
PGDATA: /var/lib/postgresql/data/
networks:
- suap_dev_net
volumes:
- postgres:/var/lib/postgresql/data
command: >
ash -c "echo host all all 0.0.0.0/0 trust > /var/lib/postgresql/data/pg_hba.conf &&
echo local all postgres trust >> /var/lib/postgresql/data/pg_hba.conf &&
echo local all all ident >> /var/lib/postgresql/data/pg_hba.conf &&
echo listen_addresses = \'*\' >> /var/lib/postgresql/data/postgresql.conf"
image: postgres:13-alpine
The commented lines on the suapdb service are the ones I hoped would work out (as I have seen in other posts) but it hasn't.
Any contribution appreciated.
Related
I'm trying to run a bash script after a Postgres container starts which 1) creates a new table within the Postgres DB, and 2) runs a copy command that dumps the contents of a csv file into the newly created table.
Currently, I'm specifying the execution of the script within my docker-compose.yml file using the "command" argument, but I find that it doesn't allow the Postgres container to succesfully start. I receive the following information from the log:
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
When I remove the "command" argument everything is fine. Here is what my docker-compose.yml files looks like now:
# docker-compose.yml
version: '3'
services:
web:
build: .
command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; uvicorn app.main:app --host 0.0.0.0'
volumes:
- .:/app
expose: # new
- 8000
environment:
- DATABASE_URL=postgresql://fastapi_traefik:fastapi_traefik#db:5432/fastapi_traefik
depends_on:
- db
labels: # new
- "traefik.enable=true"
- "traefik.http.routers.fastapi.rule=Host(`fastapi.localhost`)"
db:
image: postgres:13-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
- "/Users/theComputerPerson/:/tmp"
expose:
- 5432
environment:
- POSTGRES_USER=fastapi_traefik
- POSTGRES_PASSWORD=fastapi_traefik
- POSTGRES_DB=fastapi_traefik
command: /bin/bash -c "/tmp/newtable.sh"
traefik: # new
image: traefik:v2.2
ports:
- 8008:80
- 8081:8080
volumes:
- "./traefik.dev.toml:/etc/traefik/traefik.toml"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
volumes:
postgres_data:
It may be worth noting that I'm trying to customize some of the aspects of this FastAPI project, and to turn your attention to the development files and not the production files. Please let me know if I can provide any additional information in the comments.
You are overriding the default container image startup command.
According to PostgreSQL official container image page, you can extend initialization adding your sh scripts (or even sql files) to /docker-entrypoint-initdb.d directory.
See https://hub.docker.com/_/postgres.
This approach has a caveat that this script could not be executed.
Another approach is to override default container image command adding yours in bash style: postgres; /bin/bash -c "/tmp/newtable.sh";
I have a docker-compose that brings up the psql database as below, currently I'm trying to connect to it with pgAdmin4 (not in a docker container) and be able to view it. I've been having trouble authenticating with the DB and I don't understand why.
docker-compose
version: "3"
services:
# nginx and server also have an override, but not important for this q.
nginx:
ports:
- 1234:80
- 1235:443
server:
build: ./server
ports:
- 3001:3001 # app server port
- 9230:9230 # debugging port
env_file: .env
command: yarn dev
volumes:
# Mirror local code but not node_modules
- /server/node_modules/
- ./server:/server
database:
container_name: column-db
image: 'postgres:latest'
restart: always
ports:
- 5432:5432
environment:
POSTGRES_USER: postgres # The PostgreSQL user (useful to connect to the database)
POSTGRES_PASSWORD: root # The PostgreSQL password (useful to connect to the database)
POSTGRES_DB: postgres # The PostgreSQL default database (automatically created at first launch)
volumes:
- ./db-data/:/var/lib/postgresql/data/
networks:
app-network:
driver: bridge
I do docker-compose up then check the logs, and it says that it is ready for connections. I go to pgAdmin and enter the following:
where password is root. I then get this error:
FATAL: password authentication failed for user "postgres"
I check the docker logs and I see
DETAIL: Role "postgres" does not exist.
I'm not sure what I'm doing wrong, according to the docs the super user should be created with those specifications. Am I missing something? Been banging my head against this for an hour now. Any help is appreciated!
#jjanes solved it in a comment, I had used a mapped volume and never properly set up the db. Removed the volume and we're good to go.
Hello i get this error after i run docker-compose build up
But i get this error
postgres_1 | Error: Database is uninitialized and superuser password is not specified.
Here is a snap shot of the error!
And down below is my docker-compose.yml file
version: '3.6'
Server.js file
services:
smart-brain-api:
container_name: backend
build: ./
command: npm start
working_dir: /usr/src/smart-brain-api
ports:
- "3000:3000"
volumes:
- ./:/usr/src/smart-brain-api
#PostGres Database
postgres:
image: postgres
ports:
- "5432:5432"
You can use the POSTGRES_HOST_AUTH_METHOD environment property by making the following change to your docker-compose.yml.
db:
image: postgres:9.6-alpine
environment:
POSTGRES_DB: "db"
POSTGRES_HOST_AUTH_METHOD: "trust"
The above will solve the error.
To avoid that you can specify the followings environments variables for postgres container on your docker-compose file.
POSTGRES_PASSWORD
This environment variable is normally required for you to use the PostgreSQL image. This environment variable sets the superuser password for PostgreSQL. The default superuser is defined by the POSTGRES_USER environment variable.
POSTGRES_DB
This optional environment variable can be used to define a different name for the default database that is created when the image is first started. If it is not specified, then the value of POSTGRES_USER will be used.
For more information about Environment Variables check:
https://hub.docker.com/_/postgres
It's already mentioned in the interactive mode; how to run the container, if you don't find it, use the following:
To allow all connections without a password use:
docker run -e POSTGRES_HOST_AUTH_METHOD=trust postgres:9.6 (use the tag you need).
To specify postgres password for the superuser, use:
docker run -e POSTGRES_PASSWORD=<your_password> postgres:9.6 (use the tag you need).
You can make change to your docker-compose.yml file like in example:
db:
image: postgres:13
environment:
- "POSTGRES_HOST_AUTH_METHOD=trust"
Summing up the command on official docker site:
docker run --name <YOUR_POSTGRES_DB> -e POSTGRES_PASSWORD=<YOUR_POSTGRES_PASSWORD> -d postgres
You can make your connection using the below docker command.
docker run -e POSTGRES_PASSWORD=<your_password> postgres:9.6.
I have written a Dockerfile like this:
FROM postgres:11.2-alpine
ADD ./db/postgresql.conf /etc/postgresql/postgresql.conf
CMD ["-c", "config_file=/etc/postgresql/postgresql.conf"]
It just adds custom config location to a generic Postgres image.
Now I have the following docker-compose service description
db:
build:
context: .
dockerfile: ./db/Dockerfile
environment:
POSTGRES_PASSWORD passwordhere
POSTGRES_USER: user
POSTGRES_DB: db_name
ports:
- 5432:5432
volumes:
- ./run/db-data:/var/lib/db/data
The problem is I can no longer remotely connect to DB using these credentials if I add this Config option. Without that CMD line it works just fine.
If I prepend "postgres" in CMD it has the same effect due to the underlying script prepending it itself.
Provided all the files are where they need to be, I believe the only problem with your setup is that you've omitted an actual executable from the CMD -- specifying just the option. You need to actually run postgres:
CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
That should work!
EDIT in response to OP's first comment below
First, I did confirm that behavior doesn't change whether "postgres" is in the CMD or not. It's exactly as you said. Onward!
Then I thought there must be a problem with the particular postgresql.conf in use. If we could just figure out what the default file is.. turns out we can!
How to get the existing postgres.conf out of the postgres image
1. Create docker-compose.yml with the following contents:
version: "3"
services:
db:
image: postgres:11.2-alpine
environment:
- POSTGRES_PASSWORD=passwordhere
- POSTGRES_USER=user
- POSTGRES_DB=db_name
ports:
- 5432:5432
volumes:
- ./run/db-data:/var/lib/db/data
2. Spin up the service using
$ docker-compose run --rm --name=postgres db
3. In another terminal get the location of the file used in this release:
$ docker exec -it postgres psql --dbname=db_name --username=user --command="SHOW config_file"
config_file
------------------------------------------
/var/lib/postgresql/data/postgresql.conf
(1 row)
4. View the contents of default postgresql.conf
$ docker exec -it postgres cat /var/lib/postgresql/data/postgresql.conf
5. Replace local config file
Now all we have to do is replace the local config file ./db/postgresql.conf with the contents of the known-working-state config and modify it as necessary.
Database objects are only created once!
Database objects are only created once by the postgres container (source). So when developing the database parameters we have to remove them to make sure we're in a clean state.
Here's a nuclear (be careful!) option to
(1) remove all exited Docker containers, and then
(2) remove all Docker volumes not attached to containers:
$ docker rm $(docker ps -a -q) -f && docker volume prune -f
So now we can be sure to start from a clean state!
Final setup
Let's bring our Dockerfile back into the picture (just like you have in the question).
docker-compose.yml
version: "3"
services:
db:
build:
context: .
dockerfile: ./db/Dockerfile
environment:
- POSTGRES_PASSWORD=passwordhere
- POSTGRES_USER=user
- POSTGRES_DB=db_name
ports:
- 5432:5432
volumes:
- ./run/db-data:/var/lib/db/data
Connect to the db
Now all we have to do is build from a clean state.
# ensure all volumes are deleted (see above)
$ docker-compose build
$ docker-compose run --rm --name=postgres db
We can now (still) connect to the database:
$ docker exec -it postgres psql --dbname=db_name --username=user --command="SELECT COUNT(1) FROM pg_database WHERE datname='db_name'"
Finally, we can edit the postgres.conf from a known working state.
As per this other discussion, your CMD command only has arguments and is missing a command. Try:
CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
I'm trying to run a docker image with PostgreSQL that has a volume configured for persisting data.
docker-compose.yml
version: '3.1'
services:
db:
image: postgres
restart: always
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: example
When I start the container I see the output
fixing permissions on existing directory /var/lib/postgresql/data ... ok
and the data folder is no longer readable for me.
If I elevate myself and access the data directory I can see that the files are there. Furthermore, the command ls -ld data gives me
drwx------ 19 systemd-coredump root 4096 May 17 16:22 data
I can manually set the directory permission with sudo chmod 755 data, but that only works until I restart the container.
Why does this happen, and how can I fix it?
The other answer indeed points to the root cause of the problem, however the help page it points to does not contain a solution. Here is what I came up with to make this work for me:
start the container using your normal docker-compose file, this creates the directory with the hardcoded uid:gid (999:999)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_PASSWORD
stop the container and manually change the ownership to uid:gid you want (I'll use 1000:1000 for this example
$ docker stop postgres
$ sudo chown -R 1000:1000 ./data
Edit your docker file to add your desired uid:gid and start it up again using docker-compose (notice the user:)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
user: 1000:1000
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_password
The reason you can't just use user: from the start is that if the image runs as a different user it fails to create the data files.
On the image documentation page, it does mention a solution to add a volume to expose the /etc/passwd file as read-only in the image when providing --user option, however, that did not work for me with the latest image, as I was getting the following error. In fact none of the three proposed solutions worked for me.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
This is because of what is written in the dockerfile of the postgres image.
From line 15 to 18, you'll see that the group 999 and the user 999 are used. I'm guessing that in your host, they map respectively to systemd-coredump and root.
You need to know that whenever you use a user/group in an image, if the uid/gid exist in your host, then it will be mapped to it.
You can read the documentation on the docker hub from the postgres image here. There is a section Arbitrary --user Notes that explain how it works in the context of this image.
An easier and permanent solution would be as follows:
Add these lines to ~/.bashrc:
export UID=$(id -u)
export GID=$(id -g)
Reload your shell:
$ source ~/.bashrc
Modify your docker-compose.yml as follows:
version: "3.7"
services:
db:
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
user: "${UID}:${GID}"
...
Source
here's what i did:
services:
postgres:
image: postgres:15.1
restart: always
environment:
- POSTGRES_USER=my_user
- POSTGRES_PASSWORD=my_user
- POSTGRES_DB=my_user
user: root
ports:
- "5432:5432"
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
postgres_setup:
image: postgres:15.1
user: root
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
entrypoint: [ "bash", "-c", "chmod 750 -R /var/lib/postgresql/data && chmod 750 -R /etc/postgresql"]
depends_on:
- postgres
pgadmin4:
image: dpage/pgadmin4
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=my_user#admin.com
- PGADMIN_DEFAULT_PASSWORD=my_user
- PGADMIN_LISTEN_ADDRESS=0.0.0.0
user: root
ports:
- "5050:80"
volumes:
- /home/my_user/volumes/pgadmin/data:/var/lib/pgadmin
depends_on:
- postgres_setup
the postgres_setup container just changes permissions and then shuts down
I have been struggling with a similar issue and the answer hit me when trying to work around postgres (static uid per container, configured or 70 by default on alpine, 999 on standard image), and docker limitations (no uid translation of volumes).
The answer is to utilize Linux ACL without any changes to docker-compose.yml user - just keep the default internal container user id.
mkdir -p ./data
sudo setfacl -m u:$(id -u):rwx -R ./data/
docker-compose up -d
or
docker-compose up -d
sudo setfacl -m u:$(id -u):rwx -R ./data/
The order of creating data volume's directory does not matter and as long as ACL is set after it was created, you as a user will be able to access it recursively. You can of course add additional permissions.
To check who has access to data folder simply run:
getfacl ./data