Docker-Compose | why the containers can't reach each other? - postgresql

I have been trying to create a docker-compose file to get my web application up and running using only 'docker-compose up'. I can't make the containers reach each other, currently, I'm trying that the backend container connects to the postgres DB.
I have added a health check for the postgres container
I have been trying to add 'network_mode: host' to the postgres container but with no success.
I am using Prisma as an ORM to connect to the DB.
This is my docker-compose.yml file:
version: "2.1"
services:
##############################
# Database Container
##############################
postgres:
restart: always
container_name: db
hostname: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
ports:
- "5432:5432"
build:
dockerfile: ./database/Dockerfile
context: .
networks:
- mynet
healthcheck:
test: ["CMD-SHELL", "pg_isready -d postgres -U postgres"]
interval: 10s
timeout: 5s
retries: 5
# ##############################
# # Backend Container
# ##############################
backend:
restart: always
container_name: backend
hostname: backend
environment:
- DATABASE_URL=postgresql://postgres:postgres#localhost:5432/postgres?schema=public
build:
dockerfile: ./Dockerfile
context: ./backend
depends_on:
postgres:
condition: service_healthy
networks:
- mynet
ports:
- "3001:3000"
# ##############################
# # Frontend Container
# ##############################
# frontend:
# restart: always
# container_name: frontend
# hostname: frontend
# build:
# dockerfile: ./Dockerfile
# context: ./frontend
# ports:
# - "3000:3000"
# depends_on:
# - "backend"
networks:
mynet:
driver: bridge
That's what I am getting ( Currently trying to communicate between the backend and Postgres containers. ):
I really appreciate any help you can provide.

Just use the name of the database-service as hostname.
- DATABASE_URL=postgresql://postgres:postgres#postgres:5432/postgres?schema=public
edit
Docker containers are a little bit like a virtual machine. They have their own isolated network. When you write down the container ports (it is the ports-part of your docker-compose), then is docker able to establish a routing between your containers.
Docker container "A" can talk to container "B" with only knowing the port and the name of the container.
When you want to talk from your host to container, then you have to publish the ports. You already did it with 3001:3001.
That means for docker: All traffic on port 3001 on the host will be redirected to container on port 3001. <portOnHost>:<portOnContainer>
I recommend to learn something about the networking. Here is documentation of docker.

Related

Docker network: Connect to Postgres DB from DataHub or OpenMetadata locally

I try to run DataHub (https://datahub.io/) and / or OpenMetaData (https://open-metadata.org/) locally for testing. Both are used via docker-compose files.
For OpenMetaData I used:
version: "3.9"
volumes:
ingestion-volume-dag-airflow:
ingestion-volume-dags:
ingestion-volume-tmp:
services:
mysql:
container_name: openmetadata_mysql
image: openmetadata/db:0.13.0
restart: always
environment:
MYSQL_ROOT_PASSWORD: password
expose:
- 3306
volumes:
- ./docker-volume/db-data:/var/lib/mysql
networks:
- app_net
healthcheck:
test: mysql --user=root --password=$$MYSQL_ROOT_PASSWORD --silent --execute "use openmetadata_db"
interval: 15s
timeout: 10s
retries: 10
elasticsearch:
container_name: openmetadata_elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
environment:
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms1024m -Xmx1024m
networks:
- app_net
ports:
- "9200:9200"
- "9300:9300"
openmetadata-server:
container_name: openmetadata_server
restart: always
image: openmetadata/server:0.13.0
environment:
# OpenMetadata Server Authentication Configuration
AUTHORIZER_CLASS_NAME: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
AUTHORIZER_REQUEST_FILTER: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
AUTHORIZER_ADMIN_PRINCIPALS: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
AUTHORIZER_INGESTION_PRINCIPALS: ${AUTHORIZER_INGESTION_PRINCIPALS:-[ingestion-bot]}
AUTHORIZER_PRINCIPAL_DOMAIN: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"openmetadata.org"}
AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
AUTHORIZER_ENABLE_SECURE_SOCKET: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
AUTHENTICATION_PROVIDER: ${AUTHENTICATION_PROVIDER:-basic}
CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
AUTHENTICATION_PUBLIC_KEYS: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/config/jwks]}
AUTHENTICATION_AUTHORITY: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
AUTHENTICATION_CLIENT_ID: ${AUTHENTICATION_CLIENT_ID:-""}
AUTHENTICATION_CALLBACK_URL: ${AUTHENTICATION_CALLBACK_URL:-""}
AUTHENTICATION_JWT_PRINCIPAL_CLAIMS: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
AUTHENTICATION_ENABLE_SELF_SIGNUP: ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
# JWT Configuration
RSA_PUBLIC_KEY_FILE_PATH: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
RSA_PRIVATE_KEY_FILE_PATH: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
JWT_ISSUER: ${JWT_ISSUER:-"open-metadata.org"}
JWT_KEY_ID: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
# OpenMetadata Server Airflow Configuration
AIRFLOW_HOST: ${AIRFLOW_HOST:-http://ingestion:8080}
SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api}
AIRFLOW_AUTH_PROVIDER: ${AIRFLOW_AUTH_PROVIDER:-no-auth}
# OpenMetadata Airflow Azure SSO Configuration
OM_AUTH_AIRFLOW_AZURE_CLIENT_SECRET: ${OM_AUTH_AIRFLOW_AZURE_CLIENT_SECRET:-""}
OM_AUTH_AIRFLOW_AZURE_AUTHORITY_URL: ${OM_AUTH_AIRFLOW_AZURE_AUTHORITY_URL:-""}
OM_AUTH_AIRFLOW_AZURE_SCOPES: ${OM_AUTH_AIRFLOW_AZURE_SCOPES:-[]}
OM_AUTH_AIRFLOW_AZURE_CLIENT_ID: ${OM_AUTH_AIRFLOW_AZURE_CLIENT_ID:-""}
# OpenMetadata Airflow Google SSO Configuration
OM_AUTH_AIRFLOW_GOOGLE_SECRET_KEY_PATH: ${OM_AUTH_AIRFLOW_GOOGLE_SECRET_KEY_PATH:- ""}
OM_AUTH_AIRFLOW_GOOGLE_AUDIENCE: ${OM_AUTH_AIRFLOW_GOOGLE_AUDIENCE:-"https://www.googleapis.com/oauth2/v4/token"}
# OpenMetadata Airflow Okta SSO Configuration
OM_AUTH_AIRFLOW_OKTA_CLIENT_ID: ${OM_AUTH_AIRFLOW_OKTA_CLIENT_ID:-""}
OM_AUTH_AIRFLOW_OKTA_ORGANIZATION_URL: ${OM_AUTH_AIRFLOW_OKTA_ORGANIZATION_URL:-""}
OM_AUTH_AIRFLOW_OKTA_PRIVATE_KEY: ${OM_AUTH_AIRFLOW_OKTA_PRIVATE_KEY:-""}
OM_AUTH_AIRFLOW_OKTA_SA_EMAIL: ${OM_AUTH_AIRFLOW_OKTA_SA_EMAIL:-""}
OM_AUTH_AIRFLOW_OKTA_SCOPES: ${OM_AUTH_AIRFLOW_OKTA_SCOPES:-[]}
# OpenMetadata Airflow Auth0 SSO Configuration
OM_AUTH_AIRFLOW_AUTH0_CLIENT_ID: ${OM_AUTH_AIRFLOW_AUTH0_CLIENT_ID:-""}
OM_AUTH_AIRFLOW_AUTH0_CLIENT_SECRET: ${OM_AUTH_AIRFLOW_AUTH0_CLIENT_SECRET:-""}
OM_AUTH_AIRFLOW_AUTH0_DOMAIN_URL: ${OM_AUTH_AIRFLOW_AUTH0_DOMAIN_URL:-""}
# OpenMetadata Airflow Custom OIDC SSO Configuration
OM_AUTH_AIRFLOW_CUSTOM_OIDC_CLIENT_ID: ${OM_AUTH_AIRFLOW_CUSTOM_OIDC_CLIENT_ID:-""}
OM_AUTH_AIRFLOW_CUSTOM_OIDC_SECRET_KEY: ${OM_AUTH_AIRFLOW_CUSTOM_OIDC_SECRET_KEY:-""}
OM_AUTH_AIRFLOW_CUSTOM_OIDC_TOKEN_ENDPOINT_URL: ${OM_AUTH_AIRFLOW_CUSTOM_OIDC_TOKEN_ENDPOINT_URL:-""}
# OpenMetadata Airflow JWT Token Configuration
OM_AUTH_JWT_TOKEN: ${OM_AUTH_JWT_TOKEN:-""}
# Database configuration for MySQL
DB_DRIVER_CLASS: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
DB_SCHEME: ${DB_SCHEME:-mysql}
DB_USE_SSL: ${DB_USE_SSL:-false}
DB_USER: ${DB_USER:-openmetadata_user}
DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
DB_HOST: ${DB_HOST:-mysql}
DB_PORT: ${DB_PORT:-3306}
OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
# Airflow SSL Configurations
AIRFLOW_VERIFY_SSL: ${AIRFLOW_VERIFY_SSL:-"no-ssl"}
AIRFLOW_SSL_CERT_PATH: ${AIRFLOW_SSL_CERT_PATH:-""}
# ElasticSearch Configurations
ELASTICSEARCH_HOST: ${ELASTICSEARCH_HOST:- elasticsearch}
ELASTICSEARCH_PORT: ${ELASTICSEARCH_PORT:-9200}
ELASTICSEARCH_SCHEME: ${ELASTICSEARCH_SCHEME:-http}
ELASTICSEARCH_USER: ${ELASTICSEARCH_USER:-""}
ELASTICSEARCH_PASSWORD: ${ELASTICSEARCH_PASSWORD:-""}
# Heap OPTS Configurations
OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
expose:
- 8585
- 8586
ports:
- "8585:8585"
- "8586:8586"
depends_on:
elasticsearch:
condition: service_started
mysql:
condition: service_healthy
networks:
- app_net
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:8586/healthcheck" ]
ingestion:
container_name: openmetadata_ingestion
image: openmetadata/ingestion:0.13.0
depends_on:
elasticsearch:
condition: service_started
mysql:
condition: service_healthy
openmetadata-server:
condition: service_healthy
environment:
AIRFLOW__API__AUTH_BACKENDS: airflow.api.auth.backend.basic_auth
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/opt/airflow/dag_generated_configs"
DB_HOST: ${AIRFLOW_DB_HOST:-mysql}
DB_PORT: ${AIRFLOW_DB_PORT:-3306}
AIRFLOW_DB: ${AIRFLOW_DB:-airflow_db}
AIRFLOW_DB_SCHEME: ${AIRFLOW_DB_SCHEME:-mysql+pymysql}
DB_USER: ${AIRFLOW_DB_USER:-airflow_user}
DB_PASSWORD: ${AIRFLOW_DB_PASSWORD:-airflow_pass}
entrypoint: /bin/bash
command:
- "/opt/airflow/ingestion_dependency.sh"
expose:
- 8080
ports:
- "8080:8080"
networks:
- app_net
volumes:
- ingestion-volume-dag-airflow:/opt/airflow/dag_generated_configs
- ingestion-volume-dags:/opt/airflow/dags
- ingestion-volume-tmp:/tmp
networks:
app_net:
ipam:
driver: default
config:
- subnet: "172.16.240.0/24"
and for DataHub I followed the steps described here: https://datahubproject.io/docs/quickstart/
Everything runs locally on my Mac.
Now my problem - I have also a local postgres up and running. I can access the database via PGAdmin.
version: '3.8'
services:
db:
container_name: pg_container
image: postgres
restart: always
environment:
POSTGRES_USER: admin
POSTGRES_PASSWORD: admin
POSTGRES_DB: phd_test
ports:
- "5432:5432"
pgadmin:
container_name: pgadmin4_container
image: dpage/pgadmin4
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: admin#admin.com
PGADMIN_DEFAULT_PASSWORD: admin
ports:
- "5050:80"
In both cases DataHub & OpenMetaData I can not connect to this local Postgres DB. In both cases I get errors like for example:
'sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "127.0.0.1", port 5432 failed: Connection refused\n'
'\tIs the server running on that host and accepting TCP/IP connections?\n'
'\n'
'(Background on this error at: https://sqlalche.me/e/14/e3q8)\n',
"2022-12-04 13:58:08.971818 [exec_id=9e2a2163-24e0-4afb-8d2d-cb1b3033bc91] INFO: Failed to execute 'datahub ingest'",
I also tried different endpoints within DataHub and OpenMetadata for the Postgres like
127.0.0.1:5432
0.0.0.0:5432
pg_container:5432 (Name of the postgres container - this works in PG Admin)
172.25.0.2:5432 (Ip Address of the pg_container)
Does anybody else have a local connection between one of these tools up and running? As the errors are similar in both tools I think that it could be a docker network error, so that the postgres container can not be seen by other containers (like DataHub or OpenMetaData).
Maybe you figured it out, but the problem here is that the open metadata is running in a docker container; thus, localhost is the same open metadata container. If you are running Postgres locally you need to access the host's localhost and not your own container address therefore you need to use the host host.docker.internal.
Similarly, if you are using a docker container for Postgres and not exposing the port to your local network you need to find out the host of that container which is usually the id. For this, you can run docker network ls to see all the containers in docker bridge network and then inspect the container to see the IP address and container id's using docker inspect <NETWORK ID>.

Access Postgres database remotely that's hosted on Azure in docker container with webapi

I am new to Azure cloud services so excuse me if this is a dumb question.
I have a docker-compose file with a .Net core webapi and postgres database. I have it running on Azure as a web-app and its working (I can see when I query the API that there's data in the database). However I would like to get access to the database remotely so that I can inspect and see the data in the database via pgAdmin or something similar.
I did bind a port to my pgAdmin site in my docker-compose but it does not seem like that port is open. I've read somewhere that only port 80 and 443 can be exposed from Azure web-apps when using multi-image containers. (This docker-compose works locally 100% and I can access the pgAdmin site and see the database with all its tables).
So my question is, how do I run my web-api with my postgres database on azure and have visibility to my database?
Docker-compose file:
version: '3.8'
services:
web:
container_name: 'bootcampapi'
image: 'myimage'
build:
context: .
dockerfile: backend.dockerfile
restart: always
ports:
- 8080:80
depends_on:
postgres:
condition: service_healthy
networks:
- bootcampbackend-network
postgres:
container_name: 'postgres'
restart: always
image: 'postgres:latest'
healthcheck:
test: ["CMD-SHELL", "pg_isready"]
interval: 10s
timeout: 5s
retries: 5
environment:
- POSTGRES_USER=myusername
- POSTGRES_PASSWORD=mypassword
- POSTGRES_DB=database-name
- PGDATA=database-data
networks:
- bootcampbackend-network
ports:
- 5432:5432
volumes:
- database-data:/var/lib/postgresql/data/
pgadmin:
image: dpage/pgadmin4
ports:
- 15433:80
env_file:
- .env
depends_on:
- postgres
networks:
- bootcampbackend-network
volumes:
- database-other:/var/lib/pgadmin/
networks:
bootcampbackend-network:
driver: bridge
As you have found, App Service only listens on one port. One solution around that is to use a reverse proxy like Nginx to route the traffic to both your containers.
BTW, build, depends_on and networks are unsupported. See doc

docker springboot only connects on postgres 5432 via docker-compose

Using docker to connect springboot to postgres via docker-compose. Using port 5432 on postgres works fine, if i try an port other than that it fails
working code
spring
spring.datasource.url=jdbc:postgresql://db:5432/wwc
spring.datasource.username=wwc
spring.datasource.password=test
spring.datasource.driver-class-name=org.postgresql.Driver
docker-compose
version: '2.1'
services:
db:
container_name: db
image: postgres:9.4
ports:
- 5432:5432
volumes:
- /tmp:/var/lib/postgresql
environment:
- POSTGRES_USER=wwc
- POSTGRES_DB=wwc
- POSTGRES_PASSWORD=test
server:
container_name: spring-boot-rest-server
build:
context: .
dockerfile: Dockerfile.server
ports:
- 8080:8080
logging:
driver: json-file
depends_on:
- db
web:
container_name: nginx-web
links:
- "server:springboot"
build:
context: .
dockerfile: Dockerfile.web
ports:
- 80:80
- 8088:8088
logging:
driver: json-file
depends_on:
- server
**connection refused code **
spring
spring.datasource.url=jdbc:postgresql://db:6000/wwc
spring.datasource.username=wwc
spring.datasource.password=test
spring.datasource.driver-class-name=org.postgresql.Driver
docker-compose
version: '2.1'
services:
db:
container_name: db
image: postgres:9.4
ports:
- 6000:5432
volumes:
- /tmp:/var/lib/postgresql
environment:
- POSTGRES_USER=wwc
- POSTGRES_DB=wwc
- POSTGRES_PASSWORD=test
server:
container_name: spring-boot-rest-server
build:
context: .
dockerfile: Dockerfile.server
ports:
- 8080:8080
logging:
driver: json-file
depends_on:
- db
web:
container_name: nginx-web
links:
- "server:springboot"
build:
context: .
dockerfile: Dockerfile.web
ports:
- 80:80
- 8088:8088
logging:
driver: json-file
depends_on:
- server
error:
spring-boot-rest-server | org.postgresql.util.PSQLException: Connection to db:6000 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
What am i doing wrong?
You are confusing a bit the ports: your "db" container only exports 1 port: 5432. The 6000 that you put in your docker-compose is the port on localhost that you map to that container (db) on that port (5432).
You shouldn't even use the port mappings for the postgres container unless you want to connect from localhost which I guess you don't.
If you want to use another port than 5432 you need to extend the postgres Dockerfile and change the configuration so that postgres starts listening on a different port.
Hope this helps.
In other words: The port mapping configured in docker-compose has no relevancy to how the containers connect to each other. The mapping is only relevant when something/someone attempts to connect to your containers within the docker-compose from the outside. (Like from the localhost, as #Mihai remarked.)

Cannot connect to postico from docker-compose postgresql service

I've done a docker-compose up and been able to run my web service attached to a postgresql image. Problem is, I can't view the data on postico when I try to access the database. The name of the image is db and when i try to specify hostname to be "db" on postico before i connect, i get an error saying hostname not found. I've entered my credentials, port and database name the same way i keyed them in my docker-compose file.
Does anybody know how i can find the correct setup to connect to within the container?
version: '3.6'
services:
phoenix:
# tell docker-compose which Dockerfile it needs to build
build:
context: .
dockerfile: Dockerfile.phoenix.development
# map the port of phoenix to the local dev port
ports:
- 4000:4000
# mount the code folder inside the running container for easy development
volumes:
- ./my_app:/app
# make sure we start mongodb when we start this service
# links:
# - db
depends_on:
- db
- redis
environment:
GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
FACEBOOK_CLIENT_ID: ${FACEBOOK_CLIENT_ID}
FACEBOOK_CLIENT_SECRET: ${FACEBOOK_CLIENT_SECRET}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
go:
build:
context: .
dockerfile: Dockerfile.go.development
ports:
- 8080:8080
volumes:
- ./genesys-api:/go/src/github.com/sc4224/genesys-api
depends_on:
- db
- redis
- phoenix
db:
container_name: db
image: postgres:latest
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
volumes:
- ./data/db:/data/db
restart: always
redis:
container_name: redis
image: redis:latest
ports:
- "6379:6379"
volumes:
- ./data/redis:/data/redis
entrypoint: redis-server
restart: always
use hostname as localhost.
You can't use the hostname db outside the internal docker network. That would work in the applications running in the same network.
Since you exposed the db to run on port 5432, it's exposed via 0.0.0.0:5432->5432/tcp and therefore is accessible with localhost as host and port 5432

How to access postgres-docker container other docker container without ip address

How to access postgres-docker container other docker container without ip address?
I want to store data in postgres by using myweb. in jar given host like localhost:5432/db..
Here my compose file:
version: "3"
services:
myweb:
build: ./myweb
container_name: app
ports:
- "8080:8080"
- "9090:9090"
networks:
- front-tier
- back-tier
depends_on:
- "postgresdb"
postgresdb:
build: ./mydb
image: ppk:postgres9.5
volumes:
- dbdata:/var/lib/postgresql
ports:
- "5432:5432"
networks:
- back-tier
volumes:
dbdata: {}
networks:
front-tier:
back-tier:
Instead of localhost:5432/db.. use postgresdb:5432/db.. connection string.
By default the container has the same hostname as the service name.
Here is my minimal working example, which is connecting a java client (boxfuse/flyway) with postgres server. The most important part is the heath check, which is delaying the start of the myweb container to the time when postgres is ready to accept connections.
Note that this can be directly executed by docker-compose up, it dosen't have any other dependencies. Both the images are from docker hub.
version: '2.1'
services:
myweb:
image: boxfuse/flyway
command: -url=jdbc:postgresql://postgresdb/postgres -user=postgres -password=123 info
depends_on:
postgresdb:
condition: service_healthy
postgresdb:
image: postgres
environment:
- POSTGRES_PASSWORD=123
healthcheck:
test: "pg_isready -q -U postgres"
That is the Docker Networking problem. The solution is to use postgresdb:5432/db in place of localhost:5432/db because the two service is in the same network named back-tier and docker deamon will use name service like a DNS name to make communication between the two container. I think that my solution will help you so.