Jenkins CI with docker-compose - how to properly stop containers - docker-compose

I am trying to use docker-compose with Jenkins CI.
The main issue for me now is - how to stop all docker containers after build finish? At the same time, we need to take in mind that the same containers can be used by others.
Here is an example:
docker-compose:
version: "3"
services:
my_app:
build: .
depends_on:
- mysql
- selenium
mysql:
...
selenium:
...
Jenkinsfile:
sh 'docker-compose -f docker-compose.yml build'
sh 'docker-compose -f docker-compose.yml run --rm my_app sh -c "some-commands'
stage('Run Checks') {
parallel(
'Tests': { sh 'docker-compose -f docker-compose.yml run --rm my_app sh -c "some command"' },
'Scan': { sh 'docker-compose -f docker-compose.yml run --rm my_app sh -c "some command"' },
'Some other things': { sh 'docker-compose -f docker-compose.yml run --rm my_app sh -c "some command"' }
)
}
sh 'docker-compose -f docker-compose.yml down'
It works as a single build. But if I run two or more builds at the same time - first completed build will execute docker-compose -f docker-compose.yml down and all other running builds will fail.
If I remove docker-compose -f docker-compose.yml down string from Jenkinsfile - mysql and selenium containers will continue to work.
What is the best way to stop all containers if they are not using?
I need to stop all containers (mysql and selenium for that example) after build is completed.

I do not understand what you want to achieve in your Jenkinsfile.
1) When do you start the containers mysql and selenium ?
The command sh 'docker-compose -f docker-compose.yml run --rm my_app sh -c "some command"' will start only the service my_app, execute "some command" and remove the container.
2) Do you really need to start 3 different containers of my_app for 'Tests', 'Scan' and 'Some other things' ? If you just need to execute "some command" in parallel, consider using the command docker-compose exec.
If your main problem is to start/stop containers when running several builds at the same time, here is a Jenkinsfile that can help :
//Build the services
sh 'docker-compose -f docker-compose.yml build'
//Create and start the containers
sh 'docker-compose -f docker-compose.yml -p my_project_${BUILD_NUMBER} up -d'
stage('Run Checks') {
parallel(
//Execute "some command" on running container my_app
'Tests': { sh 'docker-compose -f docker-compose.yml -p my_project_${BUILD_NUMBER} exec my_app sh -c "some command"' },
'Scan': { sh 'docker-compose -f docker-compose.yml -p my_project_${BUILD_NUMBER} exec my_app sh -c "some command"' },
'Some other things': { sh 'docker-compose -f docker-compose.yml -p my_project_${BUILD_NUMBER} exec my_app sh -c "some command"' }
)
}
//Stop and remove containers
sh 'docker-compose -f docker-compose.yml -p my_project_${BUILD_NUMBER} down'
The option -p my_project_${BUILD_NUMBER} in the docker-compose commands will allow you (and therefor Jenkins) to distinguish the containers launched by the different builds.
For example, the build #1 will create : my_project_1_my_app, my_project_1_mysql and my_project_1_selenium.
Then, the build #2 will create : my_project_2_my_app, my_project_2_mysql and my_project_2_selenium.
Then, the build #1 will remove : my_project_1_my_app, my_project_1_mysql and my_project_1_selenium, while the containers of build #2 are still running.
And finally the build #2 will remove : my_project_2_my_app, my_project_2_mysql and my_project_2_selenium.

Related

How give service dependency in creation of another service in docker service create command?

I want to know how we give service dependency if one service depends_on another service
In odoo image,
if we want to run a postgres container we say:
$ docker run -d -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo -e POSTGRES_DB=postgres --name db postgres:14.5
and then we run odoo image service by giving:
docker run -p 8069:8069 --name odoo **--link db:db** -t odoo
Now if we write docker service cli,
Then we write postgres like this,
docker service create --name db --mode global --network ingress --publish target=5432,published=5432 -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo -e POSTGRES_DB=postgres postgres:14.5
Then how we are going to write odoo service with depends_on postgres(db) service???
Thanks In Advance!!!

How to mount postgres backup into the postgres container running

docker run -d --rm --name dummy -v postgres_volume:/root alpine tail -f /dev/null
docker cp ./data dummy:/root
docker stop dummy
docker run -p 5432:5432 -v postgres_volume:/var/lib/postgresql -e POSTGRES_PASSWORD=password postgres
The above commands giving error: mkdir: cannot create directory ‘/var/lib/postgresql’: Permission denied.
./data in the docker cp command contains the Postgres backup.
What am I missing?
You (or rather postgres) are missing permissions for the files you have copied. This should fix it:
docker run --rm -v postgres_volume:/var/lib/postgresql postgres /bin/sh -c \
'chown --recursive $(id -u postgres):$(id -g postgres) /var/lib/postgresql'

Exposed ports become unresponsive after heavy load - Docker

I run a Postgresql Docker container database in my server with the exposed port 5432. After a heavy load from users this container became unresponsive by the port.
docker run -d --env POSTGRES_PASSWORD=postgres --env POSTGRES_USER=user --env POSTGRES_DB=database -p 5432:5432 password
To resolve this I need to enter to the container, make a backup, restart the container and import the backup.
$ docker exec -it [id] sh
# pg_dump -U user dbname > dbexport.pgsql
# exit
$ docker cp [id]:/backup.pgsql ~/backup.pgsql
$ docker stop [id]
$ docker run -d --env POSTGRES_PASSWORD=postgres --env POSTGRES_USER=user --env POSTGRES_DB=database -p 5432:5432 password
$ docker exec -it [id] sh
# psql -U user database < backup.pgsql
# exit
And everything back to work until other heavy load.
Why this happens?

Docker: repository name must be lowercase

Im getting errors trying to run my docker containers. I need the postgres and redis connected to my server application.
docker pull postgres
docker rm -f syda-postgres
docker run -p 30203:5432 --name syda-postgres -e POSTGRES_PASSWORD=password POSTGRES_USER=root POSTGRES_DB=syda postgres
docker pull redis
docker rm -f syda-inmemory
docker run -d -p 30204:6379 --name syda-inmemory redis redis-server --appendonly yes
docker pull docker.url.ee/syda/server:latest
docker rm -f syda-server
docker run -d -p 30202:8080 --name syda-server --link syda-postgres:postgres --link syda-inmemory:redis \docker.url.ee/syda/server:latest
This is the error im getting:
Error: No such container: syda-postgres
docker: invalid reference format: repository name must be lowercase.
See 'docker run --help'.
Error: No such container: syda-server
docker: Error response from daemon: could not get container for syda-postgres: No such container: syda-postgres.
See 'docker run --help'.
docker run -p 30203:5432 --name syda-postgres -e POSTGRES_PASSWORD=password POSTGRES_USER=root POSTGRES_DB=syda postgres
That tries to run a container from the image named POSTGRES_USER=root with the command/arguments to the entrypoint POSTGRES_DB=syda postgres. You need to pass the -e for each variable like:
docker run -p 30203:5432 --name syda-postgres \
-e POSTGRES_PASSWORD=password -e POSTGRES_USER=root -e POSTGRES_DB=syda \
postgres
Also, note that links are deprecated, you should use a shared network for communicating between containers. This is often done with a compose file. If you need to do it from a script, you could run:
docker pull postgres
docker pull redis
docker pull docker.url.ee/syda/server:latest
docker rm -f syda-postgres
docker rm -f syda-inmemory
docker rm -f syda-server
docker network rm syda-net
docker network create syda-net
docker run -p 30203:5432 --net syda-net --name syda-postgres \
-e POSTGRES_PASSWORD=password -e POSTGRES_USER=root -e POSTGRES_DB=syda \
postgres
docker run -d -p 30204:6379 --net syda-net --name syda-inmemory \
redis redis-server --appendonly yes
docker run -d -p 30202:8080 --net syda-net --name syda-server \
docker.url.ee/syda/server:latest

Docker wait for postgresql to be running

I am using postgresql with django in my project. I've got them in different containers and the problem is that i need to wait for postgres before running django. At this time i am doing it with sleep 5 in command.sh file for django container. I also found that netcat can do the trick but I would prefer way without additional packages. curl and wget can't do this because they do not support postgres protocol.
Is there a way to do it?
I've spent some hours investigating this problem and I got a solution.
Docker depends_on just consider service startup to run another service. Than it happens because as soon as db is started, service-app tries to connect to ur db, but it's not ready to receive connections. So you can check db health status in app service to wait for connection. Here is my solution, it solved my problem. :)
Important: I'm using docker-compose version 2.1.
version: '2.1'
services:
my-app:
build: .
command: su -c "python manage.py runserver 0.0.0.0:8000"
ports:
- "8000:8000"
depends_on:
db:
condition: service_healthy
links:
- db
volumes:
- .:/app_directory
db:
image: postgres:10.5
ports:
- "5432:5432"
volumes:
- database:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
volumes:
database:
In this case it's not necessary to create a .sh file.
This will successfully wait for Postgres to start. (Specifically line 6). Just replace npm start with whatever command you'd like to happen after Postgres has started.
services:
practice_docker:
image: dockerhubusername/practice_docker
ports:
- 80:3000
command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; npm start'
depends_on:
- db
environment:
- DATABASE_URL=postgres://postgres:password#db:5432/practicedocker
- PORT=3000
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
- POSTGRES_DB=practicedocker
If you have psql you could simply add the following code to your .sh file:
RETRIES=5
until psql -h $PG_HOST -U $PG_USER -d $PG_DATABASE -c "select 1" > /dev/null 2>&1 || [ $RETRIES -eq 0 ]; do
echo "Waiting for postgres server, $((RETRIES--)) remaining attempts..."
sleep 1
done
The simplest solution is a short bash script:
while ! nc -z HOST PORT; do sleep 1; done;
./run-smth-else;
Problem with your solution tiziano is that curl is not installed by default and i wanted to avoid installing additional stuff. Anyway i did what bereal said. Here is the script if anyone would need it.
import socket
import time
import os
port = int(os.environ["DB_PORT"]) # 5432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
while True:
try:
s.connect(('myproject-db', port))
s.close()
break
except socket.error as ex:
time.sleep(0.1)
In your Dockerfile add wait and change your start command to use it:
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.7.3/wait /wait
RUN chmod +x /wait
CMD /wait && npm start
Then, in your docker-compose.yml add a WAIT_HOSTS environment variable for your api service:
services:
api:
depends_on:
- postgres
environment:
- WAIT_HOSTS: postgres:5432
postgres:
image: postgres
ports:
- "5432:5432"
This has the advantage that it supports waiting for multiple services:
environment:
- WAIT_HOSTS: postgres:5432, mysql:3306, mongo:27017
For more details, please read their documentation.
wait-for-it small wrapper scripts which you can include in your application’s image to poll a given host and port until it’s accepting TCP connections.
can be cloned in Dockerfile by below command
RUN git clone https://github.com/vishnubob/wait-for-it.git
docker-compose.yml
version: "2"
services:
web:
build: .
ports:
- "80:8000"
depends_on:
- "db"
command: ["./wait-for-it/wait-for-it.sh", "db:5432", "--", "npm", "start"]
db:
image: postgres
Why not curl?
Something like this:
while ! curl http://$POSTGRES_PORT_5432_TCP_ADDR:$POSTGRES_PORT_5432_TCP_PORT/ 2>&1 | grep '52'
do
sleep 1
done
It works for me.
I have managed to solve my issue by adding health check to docker-compose definition.
db:
image: postgres:latest
ports:
- 5432:5432
healthcheck:
test: "pg_isready --username=postgres && psql --username=postgres --list"
timeout: 10s
retries: 20
then in the dependent service you can check the health status:
my-service:
image: myApp:latest
depends_on:
kafka:
condition: service_started
db:
condition: service_healthy
source: https://docs.docker.com/compose/compose-file/compose-file-v2/#healthcheck
If the backend application itself has a PostgreSQL client, you can use the pg_isready command in an until loop. For example, suppose we have the following project directory structure,
.
├── backend
│   └── Dockerfile
└── docker-compose.yml
with a docker-compose.yml
version: "3"
services:
postgres:
image: postgres
backend:
build: ./backend
and a backend/Dockerfile
FROM alpine
RUN apk update && apk add postgresql-client
CMD until pg_isready --username=postgres --host=postgres; do sleep 1; done \
&& psql --username=postgres --host=postgres --list
where the 'actual' command is just a psql --list for illustration. Then running docker-compose build and docker-compose up will give you the following output:
Note how the result of the psql --list command only appears after pg_isready logs postgres:5432 - accepting connections as desired.
By contrast, I have found that the nc -z approach does not work consistently. For example, if I replace the backend/Dockerfile with
FROM alpine
RUN apk update && apk add postgresql-client
CMD until nc -z postgres 5432; do echo "Waiting for Postgres..." && sleep 1; done \
&& psql --username=postgres --host=postgres --list
then docker-compose build followed by docker-compose up gives me the following result:
That is, the psql command throws a FATAL error that the database system is starting up.
In short, using an until pg_isready loop (as also recommended here) is the preferable approach IMO.
There are couple of solutions as other answers mentioned.
But don't make it complicated, just let it fail-fast combined with restart: on-failure. Your service will open connection to the db and may fail at the first time. Just let it fail. Docker will restart your service until it green. Keep your service simple and business-focused.
version: '3.7'
services:
postgresdb:
hostname: postgresdb
image: postgres:12.2
ports:
- "5432:5432"
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=secret
- POSTGRES_DB=Ceo
migrate:
image: hanh/migration
links:
- postgresdb
environment:
- DATA_SOURCE=postgres://user:secret#postgresdb:5432/Ceo
command: migrate sql --yes
restart: on-failure # will restart until it's success
Check out restart policies.
None of other solution worked, except for the following:
version : '3.8'
services :
postgres :
image : postgres:latest
environment :
- POSTGRES_DB=mydbname
- POSTGRES_USER=myusername
- POSTGRES_PASSWORD=mypassword
healthcheck :
test: [ "CMD", "pg_isready", "-q", "-d", "mydbname", "-U", "myusername" ]
interval : 5s
timeout : 5s
retries : 5
otherservice:
image: otherserviceimage
depends_on :
postgres:
condition: service_healthy
Thanks to this thread: https://github.com/peter-evans/docker-compose-healthcheck/issues/16
Sleeping until pg_isready returns true unfortunately is not always reliable. If your postgres container has at least one initdb script specified, postgres restarts after it is started during it's bootstrap procedure, and so it might not be ready yet even though pg_isready already returned true.
What you can do instead, is to wait until docker logs for that instance return a PostgreSQL init process complete; ready for start up. string, and only then proceed with the pg_isready check.
Example:
start_postgres() {
docker-compose up -d --no-recreate postgres
}
wait_for_postgres() {
until docker-compose logs | grep -q "PostgreSQL init process complete; ready for start up." \
&& docker-compose exec -T postgres sh -c "PGPASSWORD=\$POSTGRES_PASSWORD PGUSER=\$POSTGRES_USER pg_isready --dbname=\$POSTGRES_DB" > /dev/null 2>&1; do
printf "\rWaiting for postgres container to be available ... "
sleep 1
done
printf "\rWaiting for postgres container to be available ... done\n"
}
start_postgres
wait_for_postgres
You can use the manage.py command "check" to check if the database is available (and wait 2 seconds if not, and check again).
For instance, if you do this in your command.sh file before running the migration, Django has a valid DB connection while running the migration command:
...
echo "Waiting for db.."
python manage.py check --database default > /dev/null 2> /dev/null
until [ $? -eq 0 ];
do
sleep 2
python manage.py check --database default > /dev/null 2> /dev/null
done
echo "Connected."
# Migrate the last database changes
python manage.py migrate
...
PS: I'm not a shell expert, please suggest improvements.
#!/bin/sh
POSTGRES_VERSION=9.6.11
CONTAINER_NAME=my-postgres-container
# start the postgres container
docker run --rm \
--name $CONTAINER_NAME \
-e POSTGRES_PASSWORD=docker \
-d \
-p 5432:5432 \
postgres:$POSTGRES_VERSION
# wait until postgres is ready to accept connections
until docker run \
--rm \
--link $CONTAINER_NAME:pg \
postgres:$POSTGRES_VERSION pg_isready \
-U postgres \
-h pg; do sleep 1; done
An example for Nodejs and Postgres api.
#!/bin/bash
#entrypoint.dev.sh
echo "Waiting for postgres to get up and running..."
while ! nc -z postgres_container 5432; do
# where the postgres_container is the hos, in my case, it is a Docker container.
# You can use localhost for example in case your database is running locally.
echo "waiting for postgress listening..."
sleep 0.1
done
echo "PostgreSQL started"
yarn db:migrate
yarn dev
# Dockerfile
FROM node:12.16.2-alpine
ENV NODE_ENV="development"
RUN mkdir -p /app
WORKDIR /app
COPY ./package.json ./yarn.lock ./
RUN yarn install
COPY . .
CMD ["/bin/sh", "./entrypoint.dev.sh"]
If you want to run it with a single line command. You can just connect to the container and check if postgres is running
docker exec -it $DB_NAME bash -c "\
until psql -h $HOST -U $USER -d $DB_NAME-c 'select 1'>/dev/null 2>&1;\
do\
echo 'Waiting for postgres server....';\
sleep 1;\
done;\
exit;\
"
echo "DB Connected !!"
Inspired by #tiziano answer and the lack of nc or pg_isready, it seems that in a recent docker python image (python:3.9 here) that curl is installed by default and I have the following check running in my entrypoint.sh:
postgres_ready() {
$(which curl) http://$DBHOST:$DBPORT/ 2>&1 | grep '52'
}
until postgres_ready; do
>&2 echo 'Waiting for PostgreSQL to become available...'
sleep 1
done
>&2 echo 'PostgreSQL is available.'
Trying with a lot of methods, Dockerfile, docker compose yaml, bash script. Only last of method help me: with makefile.
docker-compose up --build -d postgres
sleep 2
docker-compose up --build -d app