I have been trying to create a Postgres database using the following docker compose file:
version: '3'
services:
postgres:
image: "postgres"
environment:
- POSTGRES_USER=root
- POSTGRES_PASSWORD=root
- POSTGRES_DB=root
ports:
- "5432:5432"
volumes:
- ./migration/docker-database-initial.sql:/docker-entrypoint-initdb.d/docker-database-initial.sql
pgadmin-compose:
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: "kaio#mail.com"
PGADMIN_DEFAULT_PASSWORD: "******"
ports:
- "54321:80"
depends_on:
- postgres
It tries to initialize a table with this script:
create table wikis(
id serial primary key,
name varchar,
bio varchar
);
INSERT INTO wikis(name, bio) VALUES
('Name1', 'Bio1.'),
('Name2', 'Bio2.');
After the db is created and connected to the postgres container host, the table is not initialized. I've found this error log during the build process:
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/docker-database-initial.sql
psql: error: /docker-entrypoint-initdb.d/docker-database-initial.sql: Operation not permitted
Expected output:
Actual output:
I've tried enabling Full Disk Access to my terminal on MacOS Security & Privacy.
Also ran the following chmod commands in the script's directory:
✘ kaiocunha#Kaios-MacBook-Pro ~/Desktop/projects/git/go/api-go-rest ↱ master ± chmod 775 -R migration
chmod: -R: No such file or directory
✘ kaiocunha#Kaios-MacBook-Pro ~/Desktop/projects/git/go/api-go-rest ↱ master chmod 775 -R migration/docker-database-initial.sql
chmod: -R: No such file or directory
✘ kaiocunha#Kaios-MacBook-Pro ~/Desktop/projects/git/go/api-go-rest ↱ master
The whole code can be found here.
Does anyone know why this is happening?
Related
I have "dockerized" a Django/PostgreSQL app and try to connect to my database
I have 2 containers: web et db
It works but I can't connect to my postgresql database
I used to ran docker exec -it coverage_africa_db_1 psql -U postgres but I got an error
psql: error: could not connect to server: FATAL: role "postgres" does not exist
I try to 'jump' into my container by running the command docker exec -it aab213f730cd bash and try to connect using psql command...
psql -d db_dev
psql: error: could not connect to server: FATAL: role "root" does not exist
or
psql -U postgres
error: could not connect to server: FATAL: role "postgres" does not exist
in fact, none of psql options works...
.env.dev
SECRET_KEY=*************************************
DEBUG=1
DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=db_dev
SQL_USER=user
SQL_PASSWORD=user
SQL_HOST=db
SQL_PORT=5432
DATABASE=postgres
DJANGO_SETTINGS_MODULE=core.settings.dev
docker-compose.yml
version: '3.7'
services:
web:
build: ./app
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app/:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- db
db:
image: postgres:12.0-alpine
restart: always
volumes:
- postgres_data:/var/lib/postgres/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_DB=db_dev
volumes:
postgres_data:
With postgres container, this:
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=user
- POSTGRES_DB=db_dev
defines how the database is initialized. If you didn't change it, you should be able to connect as user 'user' with password 'user'.
If you did change it, then the actual values are those which were present at the first launch. After first launch those credentials are written into the database, which data is on postgres_data volume. If you want to delete the data and reinitialize database with new credentials, use docker-compose down -v.
Every time I try to build my image, I get the following error:
The server must be started by the user that owns the data directory.
The following is my docker file:
version: "3.7"
services:
db:
image: postgres
container_name: xxxxxxxxxxxx
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: $POSTGRES_DB
POSTGRES_USER: $POSTGRES_USER
POSTGRES_PASSWORD: $POSTGRES_PASSWORD
nginx:
image: nginx:latest
restart: always
container_name: xxxxxxxxxxxx-nginx
volumes:
- ./deployment/nginx:/etc/nginx
logging:
driver: none
depends_on: ["radio"]
ports:
- 8080:80
- 8081:443
radio:
build:
context: .
dockerfile: "./deployment/Dockerfile"
image: test-radio
command: './manage.py runserver 0:3000'
container_name: xxxxxxxxxxxxxxx
restart: always
depends_on: ["db"]
volumes:
- type: bind
source: ./api
target: /app/api
- type: bind
source: ./xxxxxx
target: /app/xxxxx
environment:
POSTGRES_DB: $POSTGRES_DB
POSTGRES_USER: $POSTGRES_USER
POSTGRES_PASSWORD: $POSTGRES_PASSWORD
POSTGRES_HOST: $POSTGRES_HOST
AWS_KEY_ID: $AWS_KEY_ID
AWS_ACCESS_KEY: $AWS_ACCESS_KEY
AWS_S3_BUCKET_NAME: $AWS_S3_BUCKET_NAME
networks:
default:
The image is built with the following run.sh file:
#!/usr/bin/env sh
if [ ! -f .pass ]; then
openssl rand -base64 32 > .pass
fi
#export POSTGRES_DB="xxxxxxxxxxxxxxxxx"
#export POSTGRES_USER="xxxxxxxxxxxxxx"
#export POSTGRES_PASSWORD="xxxxxxxxxxxxxxxxxxxx"
#export POSTGRES_HOST="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export POSTGRES_DB="xxxxxxxxxxxxxxxxxx"
export POSTGRES_USER="xxxxxxxxxxxxxxxxxxxx"
export POSTGRES_PASSWORD="`cat .pass`"
export POSTGRES_HOST="db"
export AWS_KEY_ID="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export AWS_ACCESS_KEY="xxxxxxxxxxxxxxxxxxxxxxxxx"
export AWS_S3_BUCKET_NAME=""
echo "Your psql password is in .pass do not commit this file."
echo "The app will be available on localhost:8080 shortly"
if [ -z "$1" ]; then
docker-compose up
else
docker-compose up $1
fi
I'm wondering if my error is being caused by attempting to use a bash script to deploy the service on a Windows machine?
Details on the issue
The behavior observed by the OP definetely comes from a UID/GID mismatch, given that the specification
volumes:
- ./postgres-data:/var/lib/postgresql/data
(which can be viewed as a docker-compose equivalent of docker run -v "$PWD/postgres-data:/var/lib/postgresql/data" …) bind-mounts the $PWD/postgres-data folder inside the container, giving access to its files as is (including owner/group metadata).
Also, note that the handling of owner/group metadata between host and containers only relies on the numeric UID and GID, not on the owner and group names.
For more information about UIDs and GIDs in a Docker context, see also that article on Medium.
Workarounds if the bind-mount is necessary
For completeness, several possible solutions to workaround the bind-mount UID-mismatch issue (including the most straightforward one that consists in changing the files' UID :) are described in this answer on StackOverflow:
How to have host and container read/write the same files with Docker?
Other solutions
Following #ParanoidPenguin's comment, you may want to use a named volume, which mainly consists in using:
the docker volume command
and/or the docker run option -v …:….
Remarks:
docker run -v PATH1:PATH2 … triggers a bind-mount of PATH1 (host) to PATH2 (container) if and only if PATH1 is absolute (i.e., starts with a /) (e.g., -v "$PWD:$PWD" is a common idiom)
docker run -v NAME:PATH2 … mounts volume NAME to PATH2 (container) if and only if NAME does not contain any / (i.e., matches regexp [a-zA-Z0-9][a-zA-Z0-9_.-]).
even if we don't run docker volume create foo beforehand by hand, docker run -v foo:/data --rm -it debian will create the named volume foo if need be.
in order to populate the files of a named volume (or respectively, backup them) you can use an ephemeral container of image debian, ubuntu or so, combining at the same time a bind-mount and a volume mount:
Add a file /home/user/bar.txt in a new volume foo
file1=/home/user/bar.txt # initial file
uid=2000 # target User-ID in the volume
gid=2000 # target Group-ID in the volume
docker pull debian
docker run -v "$file1:$file1:ro" -v foo:/data \
-e file1="$file1" -e uid="$uid" -e gid="$gid" \
--rm -it debian bash -exc \
'cp -v -- "$file1" /data/bar.txt && chown -v $uid:$gid /data/bar.txt'
docker volume ls
Backup the foo volume in a tarball
date=$(date +'%Y%m%d_%H%M%S')
back="backup_$date.tar.gz"
destdir=/home/user/backup
mkdir -p "$destdir"
docker run -v foo:/data -v "$destdir:/backup" -e back="$back" \
--rm -it debian bash -exc 'tar cvzf "/backup/$back" /data'
I created docker-compose.yml which content you can find below. I navigate to the folder where file resist and run command:
docker-compose up -d
This was shown:
Starting postgres ... done
then i run that query:
docker-compose ps
Result:
Name Command State Ports
---------------------------------------------------------
postgres docker-entrypoint.sh postgres Exit 1
Now i wanted to run some command:
docker exec -it postgres psql -h localhost -p 54320 -U robert
This is what i get:
Error response from daemon: Container ae1565a84bcf0b3662b47d4f277efd2830273554b6bcf4437129e33b31c88b35 is not running
Is my container not running or? please of support.
docker-compose.yml:
version: "3"
services:
# Create a service named db.
db:
# Use the Docker Image postgres. This will pull the newest release.
image: "postgres"
# Give the container the name my_postgres. You can changes to something else.
container_name: "postgres"
# Setup the username, password, and database name. You can changes these values.
environment:
- POSTGRES_USER=robert
- POSTGRES_PASSWORD=robert
- POSTGRES_DB=mydb
# Maps port 54320 (localhost) to port 5432 on the container. You can change the ports to fix your needs.
ports:
- "54320:5432"
# Set a volume some that database is not lost after shutting down the container.
# I used the name postgres-data but you can changed it to something else.
volumes:
- ./volumes/postgres:/var/lib/postgresql/data
Can you attempt exec
docker run -it postgres psql -h localhost -p 54320 -U robert
?
$ docker exec --help
Usage: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
Run a command in a running container
Since your container has the status exit, you can't use docker exec
Can you use this docker-compose file?
version: "3"
volumes:
postgres_app: ~
services:
# Create a service named db.
postgres:
image: "postgres"
environment:
POSTGRES_USER: robert
POSTGRES_PASSWORD: robert
POSTGRES_DB: "mydb"
volumes:
- "postgres_app:/var/lib/postgresql/data"
ports:
- "54320:5432"
restart: always
And this command docker-compose exec postgres psql -U robert -d mydb
I hope this will help!
On my computer i executed this file
I'm starting a docker swarm with a PostgreSQL image.
I want to create a user named 'numbers' on that database.
This is my docker-compose file. The .env file contains POSTGRES_USER and POSTGRES_PASSORD. If I ssh into the container hosting the postgres image, I can see the variables when executing env.
But psql --user numbers tells me that role "numbers" does not exists.
How should I pass the POSTGRES_* vars so that the correct user is created?
version: '3'
services:
postgres:
image: 'postgres:9.5'
env_file:
- ./.env
ports:
- '5432:5432'
volumes:
- 'postgres:/var/lib/postgresql/data'
deploy:
replicas: 1
networks:
- default
restart: always
This creates the postgresql user as expected.
$ docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=numbers -d postgres
When Postgres find its data directory already initialized, he does not run the initialization script. This is the check:
if [ ! -s "$PGDATA/PG_VERSION" ]; then
....
So I recommend you to manually create that user or start from scratch (removing your volume if you can afford it, loosing the data). From command line:
docker volume ls
docker volume rm <id>
I have am trying to follow this tutorial and set up a postgresql container.
I have the following script:
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$#"
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec $cmd
And the following docker-compose.yml:
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
The problem is that when I run docker-compose up I get the following error:
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
Now I have tried setting the host as database, localhost, 0.0.0.0, and even the containers IP but nothing works, I have no idea what it should be or how to debug it, I am not 100% sure how docker-compose links the containers.
do not use depends_on. try it with "links"
version: '2'
services:
server:
build: .
ports:
- 3030:3030
links:
- database
#environment could be usefull too
environment:
DATABASE_HOST: database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
for more informations https://docs.docker.com/compose/compose-file/#links
May be an old thread to answer but I have been using depends_on with the following docker-compose file
version: '3.4'
volumes:
postgres_data:
driver: local
services:
postgres:
image: postgres
volumes:
- ./postgres_data:/var/lib/postgresql:rw
- ./deployments:/opt/jboss/wildfly/standalone/deployments:rw
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
ports:
- 5432:5432
keycloak:
image: jboss/keycloak
environment:
POSTGRES_ADDR: postgres
POSTGRES_DATABASE: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: Pa55w0rd
ports:
- 8080:8080
- 9990:9990
depends_on:
- postgres
The tutorial skips over a few things, and is confusing in that it mentions the wait-for-it.sh script, but then shows a much simplified version that doesn't work if you pass hostname:port as one argument to it.
I had a crack at getting this to work and both for future me and others I will add the steps below. I did this on MacOS, and have both docker and docker-compose installed as well as nodejs.
I don't have your node app handy so I used the one as described here https://nodejs.org/de/docs/guides/nodejs-docker-webapp/
I have the following directory structure:
/src/package.json
/src/server.js
/.pgpass
/docker-compose.yml
/Dockerfile
/wait-for-postgres.sh
The contents of these files is listed below.
Steps
From the ./src directory run $ npm install (creates package-lock.json)
Fix pgpass permissions with $ chmod 600 .pgpass
Make the script executable $ chmod +x wait-for-postgres.sh
From the root directory $ docker-compose up
It will pull the postgres image and build the node app container.
When that's done it will wait for postgres and when postgres is up you'll see it ready.
Files
The src files are exactly as per the node js dockerize link above
/src/package.json
{
"name": "docker_web_app",
"version": "1.0.0",
"description": "Node.js on Docker",
"author": "First Last <first.last#example.com>",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.16.1"
}
}
/src/server.js
'use strict';
const express = require('express');
// Constants
const PORT = 8080;
const HOST = '0.0.0.0';
// App
const app = express();
app.get('/', (req, res) => {
res.send('Hello world\n');
});
app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
.pgpass
This uses the username:password postgres:postgres and is purely for development demo purposes. In the wild you will use some other method of secrets management and never ever commit a pgpass file to version control
#host:port:db:user:pass
db:5432:*:postgres:postgres
docker-compose.yml
I have added the wait-for-postgres.sh script as a managed volume, in the original question it was bundling it in with the app src which was weird.
I have also mounted the .pgpass file in the root user's home directory, which psql will look in for auto-password completion. If you don't have some method of supplying this then you'll get an error:
psql: fe_sendauth: no password supplied
Notice the command for the server container is referring to database which is a valid docker-compose internal dns name for the postgres container.
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
volumes:
- ./wait-for-postgres.sh:/usr/app/setup/wait-for-postgres.sh
- ./.pgpass:/Users/root/.pgpass
command: ["/usr/app/setup/wait-for-postgres.sh", "database", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
Dockerfile
I have modified this from the node js tutorial, pinning it to the Debian "buster" version and also installing psql which it needs for that script.
FROM node:10-buster
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get -y update - && \
apt-get -y install libpq-dev && \
apt-get -y install postgresql-client-11
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm#5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
# Bundle app source
COPY . .
EXPOSE 8080
CMD [ "node", "server.js" ]
wait-for-postgres.sh
I have modified the script very slightly because I ran the "shellcheck" linter and it complained about a few things. I realise this script is from the docker tutorial page.
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$*"
export PGPASSFILE=./pgpass
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec "$cmd"
The problem here is the host itself.
psql -h **"$host"** -U "<USER>" -c '\l'
You are passing a wrong HOSTNAME "localhost:5432" / "192.168.64.2:5432"
What I did is setup a ~/.pgpass that has
localhost:5432:DB:USER:PASSWORD
and instead of passing "localhost:5432", omit the port. Just use "localhost"
This works for me ...