Getting 'could not translate host name "db" to address' when trying to create database migration - postgresql

I'm using FastAPI and postgresql to build a simple python api backend. When I try to build the database migration file (using Alembic) this error occures:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "db" to address: Name or service not known
As I searched a lot about this problem, it seems to be because of my docker config, but I couldn't figure it out. Here is my docker-compose code:
version: '3.7'
services:
web:
build: ./src
command: sh -c "alembic upgrade head && uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000"
volumes:
- ./src/:/usr/src/app/
ports:
- 8002:8000
environment:
- DATABASE_URL=postgresql://user:pass#db/my_db
db:
image: postgres:12.1-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=pass
- POSTGRES_DB=my_db
- POSTGRES_HOST_AUTH_METHOD=trust
ports:
- 5432:5432
volumes:
postgres_data:
I'm also suspicious to my other project that is using docker in my local machine (same setup with FastAPI, postgres, and docker), and it works fine and doesn't have any similar problem.
What should I check and change to fix the problem?
P.S: I'm a beginner in docker.

Here's what's happening:
Your web container is trying to connect to db, but db is not up yet, in simple words web cannot "see" db.
The short answer: start db first: docker-compose up -d db then start web: docker-compose up web or whatever you are running.
Now, the long answer. If you proceed with the short answer, that's ok, but it's cumbersome if you ask me. You could try changing the version to 2.x eg 2.4 and adding depends_on to web, example:
version: '2.4'
services:
web:
...
depends_on:
- db
db:
image: ...
...
If you just run docker-compose up ..., docker will start db first.
Now, you may hit another problem: postgres may not be ready to receive connections. In this case you will have to wait for it to be ready. You can achieve this by retrying the db connection on error (I don't remember the exact error, you will have to check the docs) or use something like pg_isready. Assuming this is a dev environment, you can add it to your Dockerfile by installing postgresql-client and changing your cmd to:
command: >
sh -c "
until pg_isready -q -h db; do sleep 1; done
&&
alembic upgrade head
&&
uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000"

Related

Error: timeout expired on trying to connect in Docker Postgres using pgAdmin

I've been created a docker container of postgres service, but on start it and try to connect in database I get erros like I didn't defined a user and database to Postgres instance, I already tried to change the docker-compose and find the poblem but I didn't find.
Follow the attachments:
Dockerfile:
FROM wyveo/nginx-php-fpm:latest
RUN chmod -R 775 /usr/share/nginx/
RUN export pwd=pwd
docker-compose.yml:
version: '3'
services:
laravel-app_prm:
build: .
ports:
- "8099:80"
volumes:
- ${pwd}/.docker/nginx/:/usr/share/nginx
postgres_prm:
image: postgres
restart: always
environment:
- POSTGRES_USER=db_usr
- POSTGRES_PASSWORD=postgres_password
- POSTGRES_DB=db_prm
ports:
- "5432:5440"
volumes:
- ${pwd}/.docker/dbdata:/var/lib/postgresql/data/
**when I try to connect to the database directly through the container's bash, I get an error that user and database, both being inserted in the same way as defined in docker-compose.yml, do not exist.
sudo docker container <postgres_container_id> bash
psql -h localhost -U db_usr
... and so on...
And to set up connection in pgAdmin I got the container IP using:
sudo docker container inspect <postgres_container_id>
and getting the value from IPAddress atribute.

Can't see my mongo database when using mongo cmd on a Docker container

Similar to Can't connect to MongoDB container from other Docker container - but answers from this post don't work for me.
I am new to Docker. Trying to learn it on a typescript/express/mongo/mongoose api example.
What I am trying to do (and having problems with), is to use mongo cmd line on a running mongo container after it has been spun up using docker compose up. Even though I have my data nicely persisted on a Docker volume, I don't seem to be able to log into the database using cmd line.
This is my docker-compose.yml file:
version: '3.9'
services:
api:
container_name: api_ts
build: .
restart: unless-stopped
environment:
- DB_URL=mongodb://myself:pass123#mongo:27017/
ports:
- '3131:3131'
depends_on:
- mongo
links: # (seems to be needed)
- mongo
mongo:
container_name: mongo_container
image: mongo:latest
restart: always
volumes:
- mongo_dbv:/data/db
environment:
- MONGO_INITDB_ROOT_USERNAME=myself
- MONGO_INITDB_ROOT_PASSWORD=pass123
ports:
- '27017:27017'
volumes:
mongo_dbv: {}
This is my Dockerfile:
FROM node:alpine
WORKDIR /usr/src/app
COPY package*.json .
RUN npm ci
COPY . .
ENV PORT=3131
EXPOSE 3131
COPY .env ./dist
CMD ["npm", "start"]
I am running
docker compose up -d --build
After both services are ready, I do:
docker exec -it mongo_container mongo
show dbs
...and the output of the last cmd is empty
(same occurs when trying to follow the answers in the post mentioned above)
I am sure the database contains data, because I am able to verify it using REST client.
Also, I am a bit puzzled - and maybe this is somehow connected - why there is no indication, either in docker-compose.yml or in Dockerfile, of the database name which I am using. I would expect it to be part of show dbs output. Despite that, my api runs just fine.
Listing databases requires authentication
docker exec -it mongo_container mongo -u myself -p pass123
Now you can list databases
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
Note: mongo should show you warning that "mongo" shell has been superseded by "mongosh". When you use mongosh, a proper authentication error would be shown on the database listing attempt.

How to connect to a postgres database when having two docker-compose files?

First I have built an image using Dockerfile:
FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*-SNAPSHOT.jar
ADD ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/app.jar"]
as I have two docker-compose files one for production:
version: "3"
services:
app:
image: "demo:latest"
container_name: demo-production-api
restart: always
depends_on:
- "productiondb"
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://productiondb:5432/testdb
- SPRING_DATASOURCE_HIKARI_JDBC_URL=jdbc:postgresql://productiondb:5432/testdb
- SPRING_DATASOURCE_USER=tester
- SPRING_DATASOURCE_PASSWORD=test
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
ports:
- "8440:8443"
productiondb:
image: "postgres:latest"
container_name: productiondb
ports:
- "5430:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- postgres-db-production:/usr/local/var/postgres
volumes:
postgres-db-production:
and one for develop:
version: "3"
services:
app:
image: "demo:latest"
container_name: demo-develop-api
restart: always
depends_on:
- "developdb"
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://developdb:5432/testdb
- SPRING_DATASOURCE_HIKARI_JDBC_URL=jdbc:postgresql://developdb:5432/testdb
- SPRING_DATASOURCE_USER=tester
- SPRING_DATASOURCE_PASSWORD=test
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
ports:
- "8441:8443"
developdb:
image: "postgres:latest"
container_name: developdb
ports:
- "5431:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- postgres-db-develop:/usr/local/var/postgres
volumes:
postgres-db-develop:
I build both images using:
docker-compose -p demo-production-api -f docker-compose.yml up -d && docker-compose -p demo-develop-api -f docker-compose-develop.yml up -d
Now I was able to build both environments demo-develop-api and demo-production-api as well, the Spring Boot application from demo-develop-api docker image runs using the command:
docker run -it demo-develop-api
The application runs but I keep getting this error:
Caused by: java.net.UnknownHostException: productiondb
The above error happened after changing the database host in the application.properties file from localhost to productiondb first I was getting the following:
org.postgresql.util.PSQLException: Connection to localhost:5432
refused. Check that the hostname and port are correct and that the
postmaster is accepting TCP/IP connections.
Why this issue occurring or what is the cause?
How to solve this kind of issue?
As far as I see it, the issue might be that you have binded port 5430 and 5431 to 5432 and you might be having the port set to 5432 in your application.resources file. Your application should be trying to connect to the database by using either port 5430 or 5431 for production and development respectively. Please check and try this. So, make a port change in the application.resources file.
So after a long time of debugging and trials, hopefully, this is is going to save people hours, it turned out that actually, the Spring Boot application inside the container was restarting runs and crashes without any errors, which made me more confused why it is not listening or opening a port. I even doubt it that it could be a firewall or something. So basically I just tried to get a shell from the container by doing:
docker exec -it <container id or image> sh
Note: Since I am using the image openjdk:8-jdk-alpine don't do below you will not get a shell:
docker exec -it <container id or image> bash
Then I tried to get a list of open ports by doing:
netstat -tulpn | grep ":8443"
The port 8443 was not listed, I thought it could be a problem with the java program not being running, tried to execute the spring boot which executed but without any errors and the shell itself was exiting which made me more confused.
Until I have found out that container was restartig because of Spring Boot was crashing. So I enabled verbose mode by adding the below properties to application.properties then rebuild the image again:
logging.level.org.springframework.web=DEBUG
logging.level.org.hibernate=DEBUG
So I retried the last above steps where I get a shell and execute the app.jar and it turned out that the database testdb did not exist.
UPDATE: So to sum up here how I modifed my project, I created two Spring Boot Profiles for my case one for develop application-develop.properties and one for production application-production.properties:
So inside the application-develop.properties I have it mapped to a develop postgres container host and port:
spring.datasource.url=jdbc:postgresql://developdb:5432/testdb
spring.datasource.hikari.jdbc-url=jdbc:postgresql://developdb:5432/testdb
spring.datasource.username=tester
spring.jpa.generate-ddl=true
spring.datasource.password=test
spring.jpa.database-platform=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL9Dialect
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
server.port=8443
And for application-production.properties:
spring.datasource.url=jdbc:postgresql://productiondb:5432/testdb
spring.datasource.hikari.jdbc-url=jdbc:postgresql://productiondb:5432/testdb
spring.datasource.username=tester
spring.jpa.generate-ddl=true
spring.datasource.password=test
spring.jpa.database-platform=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL9Dialect
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
server.port=8443
And in the docker-compose file for develop I just define the Spring Boot profile environment variable to:
environment:
- SPRING_PROFILES_ACTIVE=develop
And for production docker-compose file I define it as below:
environment:
- SPRING_PROFILES_ACTIVE=production

Knex Migration with Docker Compose Psql

I have a problem migrating using Knex js inside my docker-compose container.
the problem is that npm run db (knex migrate:rollback && knex migrate:latest && knex seed:run) would run right before the database is even created. Is there anyway to say that I would only like to run npm run db after the database has been created?
NOTE : if I do this npm commands on the docker terminal after it has been built everything works fine. just fyi
here is my docker-compose.yml
version: '3.6'
services:
#Backend api
server:
container_name: server
build: ./
command: npm run db
working_dir: /user/src/server
ports:
- "5000:5000"
volumes:
- ./:/user/src/server
environment:
POSTGRES_URI: postgres://test:test#192.168.99.100:5432/interapp
links:
- postgres
# PostgreSQL database
postgres:
environment:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: interapp
POSTGRES_HOST: postgres
image: postgres
ports:
- "5432:5432"
and here is my Dockerfile
FROM node:10.14.0
WORKDIR /user/src/server
COPY ./ ./
RUN npm install
CMD ["/bin/bash"]
on the docker-compose.yml file, using sh (bash) for a contained environment context for your command to run in. ie. sh -c 'npm run db'
your docker-compose file would now be
secondly, use the depends_on step to wait for the database to start
services:
#Backend api
server:
container_name: server
build: ./
command: sh -c 'npm run db'
working_dir: /user/src/server
depends_on:
-postgres
ports:
- "5000:5000"
volumes:
- ./:/user/src/server
environment:
POSTGRES_URI: postgres://test:test#192.168.99.100:5432/interapp
links:
- postgres
Simply adding depends_on to server service should do the trick here.
services:
server:
depends_on:
- postgres
...
This will cause docker-compose to start postgres container before the server container. It will not however wait for postgres to be ready. In this case it shouldn't be problem, because postgres starts really quickly.
If you want something more solid, or depends_on doesn't do the trick, you can add entrypoint wrapping script to your container. See https://docs.docker.com/compose/startup-order/, where you can read more about it. There are also links to tools, so you don't have to write your own script from scratch.

Mongo restore docker container returns 'error connecting to db server'

I've been searching for an answer to this using this question here as the basis since last night, however I get this error that I can't solve:
Step 4/4 : RUN mongorestore /trapsdump/ --host mongo:27017
--->
Running in be3d84e526ab
2018-05-18T10:52:50.575+0000 Failed: error
connecting to db server: no reachable servers
I'm using Docker-compose for this. I've used Docker a few times before building containers, but am no expert, however I think everything is correctly configured. I'm using this to seed the database as we have a 'demo' collection stored on mLab.
This is my Dockerfile:
FROM node:9.11.1
RUN mkdir /src
RUN npm install nodemon -g
WORKDIR /src
ADD ./package.json /src/package.json
RUN npm install
COPY . .
EXPOSE 3000
CMD npm start
This is the docker-compose.yml:
version: "2"
services:
mongo:
container_name: mongo
image: mongo
volumes:
- ./data:/data/db
ports:
- "27017:27017"
- "11990:11990"
mongo-seed:
build: ./mongo-seed
links:
- mongo
app:
container_name: traps
restart: always
build: .
ports:
- "3000:3000"
links:
- mongo
And this is the mongo-seed/Dockerfile:
FROM mongo
RUN mongodump --uri=mongodb://xxxxxxxxxxxxx#xxxxxxxxxxxx:xxxxx/xxxxxx -o trapsdump
RUN mongorestore /trapsdump/ --host mongo:27017
If pulls down the collection from mLab and dumps it locally.
So if anyone can point out where i'm going wrong, that would be excellent.