How to connect to a postgres database when having two docker-compose files? - postgresql

First I have built an image using Dockerfile:
FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*-SNAPSHOT.jar
ADD ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/app.jar"]
as I have two docker-compose files one for production:
version: "3"
services:
app:
image: "demo:latest"
container_name: demo-production-api
restart: always
depends_on:
- "productiondb"
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://productiondb:5432/testdb
- SPRING_DATASOURCE_HIKARI_JDBC_URL=jdbc:postgresql://productiondb:5432/testdb
- SPRING_DATASOURCE_USER=tester
- SPRING_DATASOURCE_PASSWORD=test
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
ports:
- "8440:8443"
productiondb:
image: "postgres:latest"
container_name: productiondb
ports:
- "5430:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- postgres-db-production:/usr/local/var/postgres
volumes:
postgres-db-production:
and one for develop:
version: "3"
services:
app:
image: "demo:latest"
container_name: demo-develop-api
restart: always
depends_on:
- "developdb"
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://developdb:5432/testdb
- SPRING_DATASOURCE_HIKARI_JDBC_URL=jdbc:postgresql://developdb:5432/testdb
- SPRING_DATASOURCE_USER=tester
- SPRING_DATASOURCE_PASSWORD=test
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
ports:
- "8441:8443"
developdb:
image: "postgres:latest"
container_name: developdb
ports:
- "5431:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- postgres-db-develop:/usr/local/var/postgres
volumes:
postgres-db-develop:
I build both images using:
docker-compose -p demo-production-api -f docker-compose.yml up -d && docker-compose -p demo-develop-api -f docker-compose-develop.yml up -d
Now I was able to build both environments demo-develop-api and demo-production-api as well, the Spring Boot application from demo-develop-api docker image runs using the command:
docker run -it demo-develop-api
The application runs but I keep getting this error:
Caused by: java.net.UnknownHostException: productiondb
The above error happened after changing the database host in the application.properties file from localhost to productiondb first I was getting the following:
org.postgresql.util.PSQLException: Connection to localhost:5432
refused. Check that the hostname and port are correct and that the
postmaster is accepting TCP/IP connections.
Why this issue occurring or what is the cause?
How to solve this kind of issue?

As far as I see it, the issue might be that you have binded port 5430 and 5431 to 5432 and you might be having the port set to 5432 in your application.resources file. Your application should be trying to connect to the database by using either port 5430 or 5431 for production and development respectively. Please check and try this. So, make a port change in the application.resources file.

So after a long time of debugging and trials, hopefully, this is is going to save people hours, it turned out that actually, the Spring Boot application inside the container was restarting runs and crashes without any errors, which made me more confused why it is not listening or opening a port. I even doubt it that it could be a firewall or something. So basically I just tried to get a shell from the container by doing:
docker exec -it <container id or image> sh
Note: Since I am using the image openjdk:8-jdk-alpine don't do below you will not get a shell:
docker exec -it <container id or image> bash
Then I tried to get a list of open ports by doing:
netstat -tulpn | grep ":8443"
The port 8443 was not listed, I thought it could be a problem with the java program not being running, tried to execute the spring boot which executed but without any errors and the shell itself was exiting which made me more confused.
Until I have found out that container was restartig because of Spring Boot was crashing. So I enabled verbose mode by adding the below properties to application.properties then rebuild the image again:
logging.level.org.springframework.web=DEBUG
logging.level.org.hibernate=DEBUG
So I retried the last above steps where I get a shell and execute the app.jar and it turned out that the database testdb did not exist.
UPDATE: So to sum up here how I modifed my project, I created two Spring Boot Profiles for my case one for develop application-develop.properties and one for production application-production.properties:
So inside the application-develop.properties I have it mapped to a develop postgres container host and port:
spring.datasource.url=jdbc:postgresql://developdb:5432/testdb
spring.datasource.hikari.jdbc-url=jdbc:postgresql://developdb:5432/testdb
spring.datasource.username=tester
spring.jpa.generate-ddl=true
spring.datasource.password=test
spring.jpa.database-platform=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL9Dialect
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
server.port=8443
And for application-production.properties:
spring.datasource.url=jdbc:postgresql://productiondb:5432/testdb
spring.datasource.hikari.jdbc-url=jdbc:postgresql://productiondb:5432/testdb
spring.datasource.username=tester
spring.jpa.generate-ddl=true
spring.datasource.password=test
spring.jpa.database-platform=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL9Dialect
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
server.port=8443
And in the docker-compose file for develop I just define the Spring Boot profile environment variable to:
environment:
- SPRING_PROFILES_ACTIVE=develop
And for production docker-compose file I define it as below:
environment:
- SPRING_PROFILES_ACTIVE=production

Related

Docker with postgresql in flask web application (part 2)

I am building a Flask application in Python. I'm using SQLAlchemy to connect to PostgreSQL.
In the flask application, I'm using this to connect SQLAlchemy to PostgreSQL
engine = create_engine('postgresql://postgres:[mypassword]#db:5432/employee-manager-db')
And this is my docker-compose.yml
version: '3.8'
services:
backend:
build:
context: .
dockerfile: Dockerfile
ports:
- 8000:8000
volumes:
- .:/app
links:
- db:db
depends_on:
- pgadmin
db:
image: postgres:14.5
restart: always
volumes:
- .dbdata:/var/lib/postgresql
hostname: postgres
environment:
POSTGRES_PASSWORD: [mypassword]
POSTGRES_DB: employee-manager-db
pgadmin:
image: 'dpage/pgadmin4'
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: [myemail]
PGADMIN_DEFAULT_PASSWORD: [mypassword]
ports:
- "5050:80"
depends_on:
- db
I can do "docker build -t employee-manager ." to build the image. However, when I do "docker run -p 5000:5000 employee-manager" to run the image, I get an error saying
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "db" to address: Try again
Does anybody know how to fix this? Thank you so much for your help
Your containers are on different networks and that is why they don't see each other.
When you run docker-compose up, docker-compose creates a separate network and puts all the services defined inside docker-compose.yml on that network. You can see that with docker network ls.
When you run a container with docker run, it is attached to the default bridge network, which is isolated from other networks.
There are several ways to fix this, but this one will serve you in many other scenarios:
Run docker container ls and identify the name or ID of the db container that was started with docker-compose
Then run your container with:
# ID_or_name from the previous point
docker run -p 5000:5000 --network container:<ID_or_name> employee-manager
This attached the new container to the same network as your database container.
Other ways include creating a network manually and defining that network as default in the docker-compose.yml. Then you can use docker run --network <network_name> ... to attach other containers to that network.
docker run doesn't read any of the information in the docker-compose.yml file, and it doesn't see things like the Docker network that Compose automatically creates.
In your case you already have the service fully-defined in the docker-compose.yml file, so you can use Compose commands to build and restart it
docker-compose build
docker-compose up -d # will delete and recreate changed containers
(If the name of the image is important to you – maybe you're pushing to a registry – you can specify image: alongside build:. links: are obsolete and you should remove them. I'd also avoid replacing the image's content with volumes:, since this misses any setup or modification that's done in the Dockerfile and it means you're running untested code if you ever deploy the image without the mount.)

Spring Boot connection to Postgres fails when deploying with docker compose [duplicate]

I have a Java Spring Boot app which works with a Postgres database. I want to use Docker for both of them. I initially put just the Postgres in Docker, and I had a docker-compose.yml file defined like this:
version: '2'
services:
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
Then, when I issued the commands sudo dockerd and sudo docker-compose -f docker-compose.yml up, it was starting the database. I could connect using pgAdmin for example, by using localhost as server and port 5432. Then, in my Spring Boot app, inside the application.properties file I defined the following properties.
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
spring.datasource.username=sample
spring.datasource.password=sample
spring.jpa.generate-ddl=true
At this point I could run my Spring Boot app locally through Spring Suite, and it all was working fine. Then, I wanted to also add my Spring Boot app as Docker image. I first of all created a Dockerfile in my project directory, which looks like this:
FROM java:8
EXPOSE 8080
ADD /target/manager.jar manager.jar
ENTRYPOINT ["java","-jar","manager.jar"]
Then, I entered to the directory of the project issued mvn clean followed by mvn install. Next, issued docker build -f Dockerfile -t manager . followed by docker tag 9c6b1e3f1d5e myuser/manager:latest (the id is correct). Finally, I edited my existing docker-compose.yml file to look like this:
version: '2'
services:
web:
image: myuser/manager:latest
ports:
- 8080:8080
depends_on:
- db
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
But, now if I issue sudo docker-compose -f docker-compose.yml up command, the database again starts correctly, but I get errors and exit code 1 for the web app part. The problem is the connection string. I believe I have to change it to something else, but I don't know what it should be. I get the following error messages:
web_1 | 2017-06-27 22:11:54.418 ERROR 1 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
web_1 |
web_1 | org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections
Any ideas?
Each container has its own network interface with its own localhost. So change how Java points to Postgres:
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
To:
spring.datasource.url=jdbc:postgresql://db:5432/sample
db will resolve to the proper Postgres IP.
Bonus. With docker-compose you don't need to build your image by hand. So change:
web:
image: myuser/manager:latest
To:
web:
build: .
I had the same problem and I lost some time to understand and solve this problem:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
I show all the properties so that everyone understands.
application.properties:
spring.datasource.url=jdbc:postgresql://localhost:5432/testdb
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect
spring.jpa.hibernate.ddl-auto=update
docker-compose.yml:
version: "3"
services:
springapp:
build: .
container_name: springapp
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
ports:
- 8000:8080
restart: always
depends_on:
- db
db:
image: postgres
container_name: db
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=testdb
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5000:5432
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
volumes:
pgdata:
For start spring application with local database we use url localhost.
For connect to container with database we need change 'localhost' on your database service, in my case 'localhost' to 'db'.
Solution: add SPRING_DATASOURCE_URL environment in docker-compose.yml wich rewrite spring.datasource.url value for connect:
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
I hope this helps someone save his time.
You can use this.
version: "2"
services:
sample_db-postgresql:
image: postgres:9.5
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
volumes:
- sample_db:/var/lib/postgresql/data
volumes:
sample_db:
You can use ENV variable to change the db address in your docker-compose.
Dockerfile:
FROM java:8
EXPOSE 8080
ENV POSTGRES localhost
ADD /target/manager.jar manager.jar
ENTRYPOINT exec java $JAVA_OPTS -jar manager.jar --spring.datasource.url=jdbc:postgresql://$POSTGRES:5432/sample
docker-compose:
`
container_name: springapp
environment:
- POSTGRES=db`

Connect to Postgres container with PhpStorm

I would like to access my Postgres database (docker container) from PhpStorm.
docker-compose.yml
# Run docker-compose build
# Run docker-compose up
# Live long and prosper
version: '3.1'
services:
apache:
build: .docker/apache
container_name: sf-apache
ports:
- 82:80
volumes:
- .docker/config/vhosts:/etc/apache2/sites-enabled
- ${SYMFONY_APP}:/home/wwwroot/sf3
depends_on:
- php
postgres:
container_name: postgres
restart: always
image: 'postgres:12.6'
ports:
- "5432:5432"
environment:
- "POSTGRES_USER=${PGSQL_ADMIN_USER}"
- "POSTGRES_PASSWORD=${PGSQL_ADMIN_PASSWORD}"
volumes:
- ./API/var/postgres:/var/lib/postgresql/data
- .docker/postgresql/init-database.sh:/docker-entrypoint-initdb.d/init-database.sh
My PhpStorm config :
I can access to my database via docker exec -it postgres bash
If php storm is on the same host then you need to use localhost. If both phpstorm and pg is part of the same compose file , then you would use the service name since both would be in the same virtual network
I found the solution.
I have a Postgres local and a Postgres with docker. My Postgres local get the upper hand on my docker Postgres. I have killed the service and put the container service name on it.
Works perfectly.
Thanks for you help,

Docker-compose can't connect to Docker postgres container

My Postgres DB is running in a Docker container. When container is started, it says it's ready to listen on 5432.
My application container is set to depend on it.
container_name: my_postgres_db
image: library/postgres:latest
network_mode: bridge
expose:
- 5432
ports:
- 5432:5432
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=admin
- POSTGRES_DB=localdb
restart: always
The config for app:
container_name: my_test_app
depends_on:
- db
build:
context: ./
dockerfile: Dockerfile
image: my_test_app
ports:
- 8080:8080
Based on solutions to the similar questions, I changed the localhost in the DB URL to:
spring.datasource.url=jdbc:postgresql://db:5432/localdb
It causes another error = "Unknown host exception". Even if I manage to build app this way, it still doesn't work. Logs say,
Connection to localhost:5432 refused
What else am I missing?
Why is it still listening to localhost:5432 when I obviously changed it to db:5432 and gradlew clean/build it?
just change the network_mode in postgres and app to host
network_mode: host
note that this will ignore the expose option and will use the host network an containers network

docker link resolves to localhost

I'm stuck on a very strange docker problem that I've not encountered before. What I want to do is to use docker-compose to make my application available from the internet. It's currently running on a instance on DigitalOcean and I'm currently working with the following docker-compose.yml:
version: '2.2'
services:
mongodb:
image: mongo:3.4
volumes:
- ./mongo:/data/db
ports:
- "27017"
mongoadmin: # web UI for mongo
image: mongo-express
ports:
- "8081:8081"
links:
- "mongodb:mongo"
environment:
- ME_CONFIG_OPTIONS_EDITORTHEME=ambiance
- ME_CONFIG_BASICAUTH_USERNAME=user
- ME_CONFIG_BASICAUTH_PASSWORD=pass
app:
image: project/name:0.0.1
volumes:
- ./project:/usr/src/app
working_dir: /usr/src/app
links:
- "mongodb:mongodb"
environment:
- NODE_ENV=production
command: ["npm", "start"]
ports:
- "3000:3000"
Mongoadmin connects properly and is able to connect to the database, while the database itself cannot be connected to from outside the host.
The problem is that the app won't connect to the right address. It is a express server using mongoose to connect to the database. Before connecting I'm logging the url it will connect to. In my config.js I've listed mongodb://mongodb/project, but this is resolved to localhost thus resulting in MongoError: failed to connect to server [localhost:27017] on first connect. The name of the container is resolved, but not to the proper address.
I've tried to connect to the IP (in the 172.18.0.0 range) that docker addressed to the container, but that also resolved to localhost. I've looked into /etc/hosts but this does not show anything related to this. Furthermore, I'm baffled because the mongo-express container is able to connect.
I've tried changing the name of the container, thinking it might be block for some reason due to previous runs or something like that, but this did not resolve the issue
I've tried both explicit links and implicit using dockers internal DNS resolve, but both did not work.
When binding port 27017 to localhost it is able to connect, but because of security and easy configuration via environment variables, I rather have the mongodb instance not bound to localhost.
I've also tried to run this on my local machine and that works as expected, being that both mongoadmin and app are able to connect to the mongodb container. My localmachine runs Docker version 1.12.6, build 78d1802, while the VPS runs on Docker version 17.06.2-ce, build cec0b72, thus a newer version.
Could this be a newly introduced bug? Or am I missing something else? Any help would be appreciated.
Your docker-compose file seems not have linked the app and mongodb container.
You have this:
app:
image: project/name:0.0.1
volumes:
- ./project:/usr/src/app
working_dir: /usr/src/app
environment:
- NODE_ENV=production
command: ["npm", "start"]
ports:
- "3000:3000"
While I think it should be this:
app:
image: project/name:0.0.1
volumes:
- ./project:/usr/src/app
working_dir: /usr/src/app
links:
- "mongodb:mongodb"
environment:
- NODE_ENV=production
command: ["npm", "start"]
ports:
- "3000:3000"