I am creating a dockercompose file.
I want to set up a project with this file. This dockercompose will generate 2 docker containers: One for mysql and the other for apache2.
What i want to do is to automated everything and i want to generate an unique mysql password. Is it possible to do that ?
Thanks
You can set a random password for mysql via the environment variable MYSQL_RANDOM_ROOT_PASSWORD.
MYSQL_RANDOM_ROOT_PASSWORD
This is an optional variable. Set to yes to generate a random initial password for the root user (using pwgen). The generated root password will be printed to stdout (GENERATED ROOT PASSWORD: .....).
https://github.com/mysql/mysql-docker#mysql_random_root_password
Docker Compose Environment Variables:
https://docs.docker.com/compose/environment-variables/
Edit: Alternative Method - Env Injection
This is a sample alternative strategy that would allow you to use a random password for both MySQL and Django (or any other image).
You will need to install pwgen on your host system with brew, apt, or yum, and then generate a password setting it to an environment variable. You can rerun this anytime to change the password.
export DB_PASSWORD=`pwgen -Bs1 12`
Simplified docker-compose.yml:
version: '2'
services:
db:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
MYSQL_DATABASE: db_name
MYSQL_USER: user
MYSQL_PASSWORD: user_password
web:
image: django
volumes:
- ./django_admin.py:/app
expose:
- "8000"
links:
- db
environment:
DB_PASSWORD: ${DB_PASSWORD}
In your django_admin.py, you can now use the DB_PASSWORD env variable:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db_name',
'USER': 'root',
'PASSWORD': os.environ.get('DB_PASSWORD', ''),
'HOST': 'host'
}
}
Related
I'm building an app running on NodeJS using postgresql.
I'm using SequelizeJS as ORM.
To avoid using real postgres daemon and having nodejs on my own device, i'm using containers with docker-compose.
when I run docker-compose up
it starts the pg database
database system is ready to accept connections
and the nodejs server.
but the server can't connect to database.
Error: connect ECONNREFUSED 127.0.01:5432
If I try to run the server without using containers (with real nodejs and postgresd on my machine) it works.
But I want it to work correctly with containers. I don't understand what i'm doing wrong.
here is the docker-compose.yml file
web:
image: node
command: npm start
ports:
- "8000:4242"
links:
- db
working_dir: /src
environment:
SEQ_DB: mydatabase
SEQ_USER: username
SEQ_PW: pgpassword
PORT: 4242
DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
volumes:
- ./:/src
db:
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_USER: username
POSTGRES_PASSWORD: pgpassword
Could someone help me please?
(someone who likes docker :) )
Your DATABASE_URL refers to 127.0.0.1, which is the loopback adapter (more here). This means "connect to myself".
When running both applications (without using Docker) on the same host, they are both addressable on the same adapter (also known as localhost).
When running both applications in containers they are not both on localhost as before. Instead you need to point the web container to the db container's IP address on the docker0 adapter - which docker-compose sets for you.
Change:
127.0.0.1 to CONTAINER_NAME (e.g. db)
Example:
DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
to
DATABASE_URL: postgres://username:pgpassword#db:5432/mydatabase
This works thanks to Docker links: the web container has a file (/etc/hosts) with a db entry pointing to the IP that the db container is on. This is the first place a system (in this case, the container) will look when trying to resolve hostnames.
For further readers, if you're using Docker desktop for Mac use host.docker.internal instead of localhost or 127.0.0.1 as it's suggested in the doc. I came across same connection refused... problem. Backend api-service couldn't connect to postgres using localhost/127.0.0.1. Below is my docker-compose.yml and environment variables as a reference:
version: "2"
services:
api:
container_name: "be"
image: <image_name>:latest
ports:
- "8000:8000"
environment:
DB_HOST: host.docker.internal
DB_USER: <your_user>
DB_PASS: <your_pass>
networks:
- mynw
db:
container_name: "psql"
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_DB: <your_postgres_db_name>
POSTGRES_USER: <your_postgres_user>
POSTGRES_PASS: <your_postgres_pass>
volumes:
- ~/dbdata:/var/lib/postgresql/data
networks:
- mynw
If you send database vars separately. You can assign a database host.
DB_HOST=<POSTGRES_SERVICE_NAME> #in your case "db" from docker-compose file.
I had two containers one called postgresdb, and another call node
I changed my node queries.js from:
const pool = new Pool({
user: 'postgres',
host: 'localhost',
database: 'users',
password: 'password',
port: 5432,
})
To
const pool = new Pool({
user: 'postgres',
host: 'postgresdb',
database: 'users',
password: 'password',
port: 5432,
})
All I had to do was change the host to my container name ["postgresdb"] and that fixed this for me. I'm sure this can be done better but I just learned docker compose / node.js stuff in the last 2 days.
If none of the other solutions worked for you, consider manual wrapping of PgPool.connect() with retry upon having ECONNREFUSED:
const pgPool = new Pool(pgConfig);
const pgPoolWrapper = {
async connect() {
for (let nRetry = 1; ; nRetry++) {
try {
const client = await pgPool.connect();
if (nRetry > 1) {
console.info('Now successfully connected to Postgres');
}
return client;
} catch (e) {
if (e.toString().includes('ECONNREFUSED') && nRetry < 5) {
console.info('ECONNREFUSED connecting to Postgres, ' +
'maybe container is not ready yet, will retry ' + nRetry);
// Wait 1 second
await new Promise(resolve => setTimeout(resolve, 1000));
} else {
throw e;
}
}
}
}
};
(See this issue in node-postgres for tracking.)
As mentioned here.
Each container can now look up the hostname web or db and get back the appropriate container’s IP address. For example, web’s application code could connect to the URL postgres://db:5432 and start using the Postgres database.
It is important to note the distinction between HOST_PORT and CONTAINER_PORT. In the above example, for db, the HOST_PORT is 8001 and the container port is 5432 (postgres default). Networked service-to-service communication uses the CONTAINER_PORT. When HOST_PORT is defined, the service is accessible outside the swarm as well.
Within the web container, your connection string to db would look like postgres://db:5432, and from the host machine, the connection string would look like postgres://{DOCKER_IP}:8001.
So DATABASE_URL should be postgres://username:pgpassword#db:5432/mydatabase
I am here with a tiny modification about handle this.
As Andy say in him response.
"you need to point the web container to the db container's"
And taking in consideration the official documentation about docker-compose link's
"Links are not required to enable services to communicate - by default, any service can reach any other service at that service’s name."
Because of that, you can keep your docker_compose.yml in this way:
docker_compose.yml
version: "3"
services:
web:
image: node
command: npm start
ports:
- "8000:4242"
# links:
# - db
working_dir: /src
environment:
SEQ_DB: mydatabase
SEQ_USER: username
SEQ_PW: pgpassword
PORT: 4242
# DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
DATABASE_URL: "postgres://username:pgpassword#db:5432/mydatabase"
volumes:
- ./:/src
db:
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_USER: username
POSTGRES_PASSWORD: pgpassword
But it is a kinda cool way to be verbose while we are coding. So, your approach it is nice.
I have created a docker image with this command docker compose up -d
where I was able to load pgAdmin instance in http://localhost:5050/browser/
create a database and table in the same , credentials are working properly.
However when I start to run my main spring boot application CustomerApplication it fails with below error >
org.postgresql.util.PSQLException: FATAL: la autentificación password falló para el usuario «amigoscode» (pgjdbc: autodetected server-encoding to be ISO-8859-1, if the message is not readable, please check database logs and/or host, port, dbname, user, password, pg_hba.conf)
I do not know what is wrong, my credentials are correct.
what seems to be the issue?
below are application.yml and docker-compose.yml
docker-compose.yml
services:
postgres:
container_name: postgres
image: postgres
environment:
POSTGRES_USER: amigoscode
POSTGRES_PASSWORD: password
PGDATA: /data/postgres
volumes:
- postgres:/data/postgres
ports:
- "5432:5432"
networks:
- postgres
restart: unless-stopped
pgadmin:
container_name: pgadmin
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL:-pgadmin4#pgadmin.org}
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD:-admin}
PGADMIN_CONFIG_SERVER_MODE: 'False'
volumes:
- pgadmin:/var/lib/pgadmin
ports:
- "5050:80"
networks:
- postgres
restart: unless-stopped
networks:
postgres:
driver: bridge
volumes:
postgres:
pgadmin:
application.yml
server:
port: 8080
spring:
application:
name: customer
datasource:
username: amigoscode
url: jdbc:postgresql://localhost:5432/customer
password: password
jpa:
hibernate:
ddl-auto: create-drop
properties:
hibernate:
dialect: org.hibernate.dialect.PostgreSQLDialect
format_sql: true
show-sql: true
customer table Script
CREATE DATABASE customer
WITH
OWNER = amigoscode
ENCODING = 'UTF8'
LC_COLLATE = 'en_US.utf8'
LC_CTYPE = 'en_US.utf8'
TABLESPACE = pg_default
CONNECTION LIMIT = -1;
Since I have a postgres instance installed and running in my local (port 5432), the microservice customer was trying to connect to that instance, not the one from docker which was using the same port.
the solution was to change the url port from application.yml
url: jdbc:postgresql://localhost:5433/customer
and the port of postgres from docker-compose.yml
from
ports:
- "5432:5432"
to
ports:
- "5433:5432"
so microservice connects to the postgres instance in the docker image, not the local one
then re-run docker command docker compose up -d
run the CustomerApplication (SprinbgootApplication) and this time
application starts up nice and smoothly by creating the customer table.
When you created the role amigoscode did you actually set a password for it?
CREATE ROLE amigoscode WITH LOGIN PASSWORD 'password';
You may have logged into pgadmin as the postgres superuser, created the role amigoscode and never set a password. The encoding issue looks similar to the one specified here which appears when the supplied password is incorrect. It looks like that pgjdbc bug was fixed in 2014, but has possibly regressed in more modern versions >= 42.2.x as there is currently a similar unresolved issue.
Other likely alternatives:
SPRING_DATASOURCE_PASSWORD environment variable is set to something else.
You also have an application.properties file which has a different password that will take precedence over the one in application.yml.
You have a different password specified in any of the property sources that spring checks before application.yml (discussion of precedence here)
The same happened to me, the problem is that i was running another instance that was using the port 5432 (postgresql server and docker), I was using dbeaver and it was trying to connect to that instance instead the one that i wanted to use.
Check which applications are using the port and close the ones that you don't need and try again to connect to the database.
I need to setup the Keycloak docker server with the External postgres Database connection URL.
Here's my current yaml file content which is working with POstgres docker container image as mentioned
version: '3'
volumes:
postgres_data:
driver: local
services:
postgres:
image: postgres:11
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
ports:
- 5433:5432
keycloak:
image:jboss/keycloak:latest
environment:
DB_VENDOR: POSTGRES
DB_ADDR: postgres
DB_DATABASE: keycloak
DB_USER: keycloak
DB_SCHEMA: public
DB_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: password
KEYCLOAK_LOGLEVEL: DEBUG
ROOT_LOGLEVEL: DEBUG
ports:
- 8080:8080
- 8443:8443
depends_on:
- postgres
I checked the official documentation for passing external DB connection URL.
But exactly didn't get what changes will be needed in YAML file
ref: https://hub.docker.com/r/jboss/keycloak/
I tried removing the Postgres and depends_on section from services and passed the Database connection details in Kecyloak environment section in yaml but it did not worked for me
Can anyone suggest the correct YAML file changes to use PostgresDB connection URL
Thank You.
Docker containers can see each other by their service name, so here service name postgres is actually the connection url for keycloak container.
version: '3'
volumes:
postgres_data:
driver: local
services:
postgres: # Service name
image: postgres:11
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
ports:
- 5433:5432
keycloak:
image: jboss/keycloak:latest
environment:
DB_VENDOR: POSTGRES
DB_ADDR: postgres # <<< This is the address, change it to your external db ip/domain
DB_DATABASE: keycloak
DB_USER: keycloak
DB_SCHEMA: public
DB_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: password
KEYCLOAK_LOGLEVEL: DEBUG
ROOT_LOGLEVEL: DEBUG
ports:
- 8080:8080
- 8443:8443
depends_on:
- postgres
I'm building an app running on NodeJS using postgresql.
I'm using SequelizeJS as ORM.
To avoid using real postgres daemon and having nodejs on my own device, i'm using containers with docker-compose.
when I run docker-compose up
it starts the pg database
database system is ready to accept connections
and the nodejs server.
but the server can't connect to database.
Error: connect ECONNREFUSED 127.0.01:5432
If I try to run the server without using containers (with real nodejs and postgresd on my machine) it works.
But I want it to work correctly with containers. I don't understand what i'm doing wrong.
here is the docker-compose.yml file
web:
image: node
command: npm start
ports:
- "8000:4242"
links:
- db
working_dir: /src
environment:
SEQ_DB: mydatabase
SEQ_USER: username
SEQ_PW: pgpassword
PORT: 4242
DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
volumes:
- ./:/src
db:
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_USER: username
POSTGRES_PASSWORD: pgpassword
Could someone help me please?
(someone who likes docker :) )
Your DATABASE_URL refers to 127.0.0.1, which is the loopback adapter (more here). This means "connect to myself".
When running both applications (without using Docker) on the same host, they are both addressable on the same adapter (also known as localhost).
When running both applications in containers they are not both on localhost as before. Instead you need to point the web container to the db container's IP address on the docker0 adapter - which docker-compose sets for you.
Change:
127.0.0.1 to CONTAINER_NAME (e.g. db)
Example:
DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
to
DATABASE_URL: postgres://username:pgpassword#db:5432/mydatabase
This works thanks to Docker links: the web container has a file (/etc/hosts) with a db entry pointing to the IP that the db container is on. This is the first place a system (in this case, the container) will look when trying to resolve hostnames.
For further readers, if you're using Docker desktop for Mac use host.docker.internal instead of localhost or 127.0.0.1 as it's suggested in the doc. I came across same connection refused... problem. Backend api-service couldn't connect to postgres using localhost/127.0.0.1. Below is my docker-compose.yml and environment variables as a reference:
version: "2"
services:
api:
container_name: "be"
image: <image_name>:latest
ports:
- "8000:8000"
environment:
DB_HOST: host.docker.internal
DB_USER: <your_user>
DB_PASS: <your_pass>
networks:
- mynw
db:
container_name: "psql"
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_DB: <your_postgres_db_name>
POSTGRES_USER: <your_postgres_user>
POSTGRES_PASS: <your_postgres_pass>
volumes:
- ~/dbdata:/var/lib/postgresql/data
networks:
- mynw
If you send database vars separately. You can assign a database host.
DB_HOST=<POSTGRES_SERVICE_NAME> #in your case "db" from docker-compose file.
I had two containers one called postgresdb, and another call node
I changed my node queries.js from:
const pool = new Pool({
user: 'postgres',
host: 'localhost',
database: 'users',
password: 'password',
port: 5432,
})
To
const pool = new Pool({
user: 'postgres',
host: 'postgresdb',
database: 'users',
password: 'password',
port: 5432,
})
All I had to do was change the host to my container name ["postgresdb"] and that fixed this for me. I'm sure this can be done better but I just learned docker compose / node.js stuff in the last 2 days.
If none of the other solutions worked for you, consider manual wrapping of PgPool.connect() with retry upon having ECONNREFUSED:
const pgPool = new Pool(pgConfig);
const pgPoolWrapper = {
async connect() {
for (let nRetry = 1; ; nRetry++) {
try {
const client = await pgPool.connect();
if (nRetry > 1) {
console.info('Now successfully connected to Postgres');
}
return client;
} catch (e) {
if (e.toString().includes('ECONNREFUSED') && nRetry < 5) {
console.info('ECONNREFUSED connecting to Postgres, ' +
'maybe container is not ready yet, will retry ' + nRetry);
// Wait 1 second
await new Promise(resolve => setTimeout(resolve, 1000));
} else {
throw e;
}
}
}
}
};
(See this issue in node-postgres for tracking.)
As mentioned here.
Each container can now look up the hostname web or db and get back the appropriate container’s IP address. For example, web’s application code could connect to the URL postgres://db:5432 and start using the Postgres database.
It is important to note the distinction between HOST_PORT and CONTAINER_PORT. In the above example, for db, the HOST_PORT is 8001 and the container port is 5432 (postgres default). Networked service-to-service communication uses the CONTAINER_PORT. When HOST_PORT is defined, the service is accessible outside the swarm as well.
Within the web container, your connection string to db would look like postgres://db:5432, and from the host machine, the connection string would look like postgres://{DOCKER_IP}:8001.
So DATABASE_URL should be postgres://username:pgpassword#db:5432/mydatabase
I am here with a tiny modification about handle this.
As Andy say in him response.
"you need to point the web container to the db container's"
And taking in consideration the official documentation about docker-compose link's
"Links are not required to enable services to communicate - by default, any service can reach any other service at that service’s name."
Because of that, you can keep your docker_compose.yml in this way:
docker_compose.yml
version: "3"
services:
web:
image: node
command: npm start
ports:
- "8000:4242"
# links:
# - db
working_dir: /src
environment:
SEQ_DB: mydatabase
SEQ_USER: username
SEQ_PW: pgpassword
PORT: 4242
# DATABASE_URL: postgres://username:pgpassword#127.0.0.1:5432/mydatabase
DATABASE_URL: "postgres://username:pgpassword#db:5432/mydatabase"
volumes:
- ./:/src
db:
image: postgres
ports:
- "5432:5432"
environment:
POSTGRES_USER: username
POSTGRES_PASSWORD: pgpassword
But it is a kinda cool way to be verbose while we are coding. So, your approach it is nice.
What I'm trying to do: connect to postgres db with a FastAPI app via common docker-compose file. It used to work until I changed postgres configuration from default.
I'm aware this is a common problem and I tried following every tip I found so far.
I did double check spelling of evn variables. I did remove volumes, images, networks and rebuild it from scratch multiple times by now.
Here are relevant parts of docker-compose
version: "3.4"
networks:
internal:
external: false
services:
db:
image: postgres:11
ports:
- "5432:5432"
environment:
- POSTGRES_USER=test_user
- POSTGRES_PASSWORD=test_pass
- POSTGRES_DB=test_db
networks:
- internal
my_app:
depends_on:
- db
networks:
- internal
Here's how I present db to app in code.
DATABASE_URL = "postgres://test_user:test_pass#db:5432/test_db"
I continue to get 'password authentication failed for user "test_user" / Role "test_user" does not exist. / Connection matched pg_hba.conf line 95: "host all all all md5"
What did I manage to miss?
For those who find it, the problem was actually in how I tried to provide env variables.
environment:
- POSTGRES_USER: test_user
- POSTGRES_PASSWORD: test_pass
- POSTGRES_DB: test_db
works