I try to configure continuous testing/integration with odoo and postgres docker container.
But I stuck on a problem, Gitlab CI can't do any operations on docker postgres.
My goal is to put a database template into a postgres container after run it and before testing.
I tried to use ssh executor after shell executor, but I always stuck on the same problem.
Notice that all commands here can be complete on the runner without problems, I test it.
I wrote this yml files:
variables:
# Configure postgres service (https://hub.docker.com/_/postgres/)
POSTGRES_DB: db
POSTGRES_USER: odoo
POSTGRES_PASSWORD: odoo
before_script:
# Pull container version
- docker pull postgres:9.5
- docker pull odoo:8.0
after_script:
# Remove all used container
- docker stop $(docker ps -a -q) && docker rm $(docker ps -aq)
stages:
- prepare
job1:
stage: prepare
# prepare postgres db
script:
#Launch postgres container
- docker run -d -e POSTGRES_USER=$POSTGRES_USER -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD --name db postgres:9.5
# Copy and restore db template
- docker cp /home/myuser/odoov8_test.dump db:/home
- docker exec -i db su -s /bin/sh - postgres -c "createdb odoov8_test && pg_restore -d odoov8_test --no-owner --verbose /home/odoov8_test.dump"
# launch odoo with own addons folder (/own/path/to/addons:/mnt/extra-addons) testdatabase (-d) module to install without space with all dependances (-i) test unable (--test-enable) and stop after init (--stop-after-init)
- docker run -v $CI_PROJECT_DIR:/mnt/extra-addons -p 8069:8069 --name odoo --link db:db -t odoo:8.0 -- -d odoov8_test.dump -i crm,sale --test-enable --stop-after-init
I got this result:
[0KRunning with gitlab-ci-multi-runner 1.11.2 (0489844)
on Test docker odoo (7fafb15a)
[0;m[0KUsing Shell executor...
[0;mRunning on debian-8-clean...
[32;1mFetching changes...[0;m
HEAD est maintenant à 7d196ea Update .gitlab-ci.yml
Depuis https://myserver.com/APSARL/addons-ext
7d196ea..47591ac master -> origin/master
[32;1mChecking out 47591ac6 as master...[0;m
[32;1mSkipping Git submodules setup[0;m
[32;1m$ docker pull postgres:9.5[0;m
9.5: Pulling from library/postgres
Digest: sha256:751bebbc12716d7d9818678e91cbec8138e52dc4a084f0e81c58cd8b419930e5
Status: Image is up to date for postgres:9.5
[32;1m$ docker pull odoo:8.0[0;m
8.0: Pulling from library/odoo
Digest: sha256:9deda039e0df28aaf515001dd1606ab74a16ed25504447edc2912bca9019cd43
Status: Image is up to date for odoo:8.0
[32;1m$ docker run -d -e POSTGRES_USER=$POSTGRES_USER -e POSTGRES_PASSWORD=$POSTGRES_PASSWORD --name db postgres:9.5[0;m
60a0c75fd55e953e6a25a3cc0f13093ec2f1ee96bfb8384ac19d00f740dd1d4e
[32;1m$ docker cp /home/myuser/odoov8_test.dump db:/home
[0;m[32;1m$ docker exec -i db su -s /bin/sh - postgres -c "createdb odoov8_test && pg_restore -d odoov8_test --no-owner --verbose /home/odoov8_test.dump"
[0;m
createdb: could not connect to database template1: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
[32;1mRunning after script...[0;m
[32;1m$ docker stop $(docker ps -a -q) && docker rm $(docker ps -aq)[0;m
60a0c75fd55e
60a0c75fd55e
[31;1mERROR: Job failed: exit status 1
[0;m
Related
I have a simple docker-compose.yml & associated Dockerfiles that give me a simple dev and prod environment for a nginx-uvicorn-django-postgres stack. I want to add an optional 'backup' container that just runs cron to periodically connect to the 'postgres' container.
# backup container - derived from [this blog][1]
ARG DOCKER_REPO
ARG ALPINE_DOCKER_IMAGE # ALPINE
ARG ALPINE_DOCKER_TAG # LATEST
FROM ${DOCKER_REPO}${ALPINE_DOCKER_IMAGE}:${ALPINE_DOCKER_TAG}
ARG DB_PASSWORD
ARG DB_HOST # "db"
ARG DB_PORT # "5432"
ARG DB_NAME # "ken"
ARG DB_USERNAME # "postgres"
ENV PGPASSWORD=${DB_PASSWORD} HOST=${DB_HOST} PORT=${DB_PORT} PSQL_DB_NAME=${DB_NAME} \
USERNAME=${DB_USERNAME}
RUN printenv
RUN mkdir /output && \
mkdir /output/backups && \
mkdir /scripts && \
chmod a+x /scripts
COPY ./scripts/ /scripts/
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/15min/${DB_NAME}_15
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/daily/${DB_NAME}_day
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/weekly/${DB_NAME}_week
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/monthly/${DB_NAME}_month
RUN apk update && \
apk upgrade && \
apk add --no-cache postgresql-client && \
chmod a+x /etc/periodic/15min/${DB_NAME}_15 && \
chmod a+x /etc/periodic/daily/${DB_NAME}_day && \
chmod a+x /etc/periodic/weekly/${DB_NAME}_week && \
chmod a+x /etc/periodic/monthly/${DB_NAME}_month
The django container is derived from the official Python image and connects (through psycopg2) with values (as ENV value) for host, dbname, username, password and port. The 'backup' container has these same values, but I get this error from the command line:
> pg_dump --host="$HOST" --port="$PORT" --username="$USERNAME" --dbname="$PSQL_DB_NAME"
pg_dump: error: could not translate host name "db" to address: Name does not resolve
Is Alpine missing something relevant that is present in the official Python?
Edit:
I am running with a system of shell scripts that take care of housekeeping for different configurations. so
> ./ken.sh dev_server
will set up the environment variables and then run docker-compose for the project and the containers
docker-compose.yml doesn't explicitly create a network.
I don't know what "db" should resolve to beyond just 'db://'? - its what the django container gets and it is able to resolve a connection to the 'db' service.
service:
db:
image: ${DOCKER_REPO}${DB_DOCKER_IMAGE}:${DB_DOCKER_TAG} #postgres: 14
container_name: ${PROJECT_NAME}_db
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- PGPASSWORD
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
command: ["postgres", "-c", "log_statement=all"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h db"]
interval: 2s
timeout: 5s
retries: 25
This is the 'dev_server' script run by the parent ken.sh script
function dev_server() {
trap cleanup EXIT
wait_and_launch_browser &
docker-compose -p "${PROJECT_NAME}" up -d --build db nginx web pgadmin backup
echo "Generate static files and copy them into static and file volumes."
source ./scripts/generate_static_files.sh
docker-compose -p "${PROJECT_NAME}" logs -f web nginx backup
}
Update: Worked through "Reasons why docker containers can't talk to each other" and found that all the containers are on a ken_default network, from 170.20.0.2 to 170.20.0.6.
I can docker exec ken_backup backup ken_db -c2, but not from db to backup, because the db container doesn't include ping.
From a shell on backup I cannot ping ken_db - ken_db doesn't resolve, nor does 'db'.
I can't make much of that and I'm not sure what to try next.
You are running the backup container as a separate service.
Docker-compose creates a unique network for each service (docker-compose.yml file).
You need to get the DB and your backup container on the same docker network.
See this post
This question already has answers here:
How to create User/Database in script for Docker Postgres
(9 answers)
Closed 6 months ago.
I'm feeling a bit stupid, but...
All I want is a dockerfile that allows me to configure the db and the user to replace a script like
docker run --name $CONTAINER_NAME \
-p 5432:5432 \
-e POSTGRES_PASSWORD=mypass \
-e POSTGRES_DB=mydb \
-d postgres:13.5-alpine
###
### artificial wait here
###
docker exec $CONTAINER_NAME bash -c "psql --username postgres --command \"CREATE USER foo WITH SUPERUSER ENCRYPTED PASSWORD 'bar';\""
I.e. something like this (I thought):
FROM postgres:13.5-alpine
EXPOSE 5432
ENV POSTGRES_DB mydb
ENV POSTGRES_PASSWORD mypass
CMD psql --username postgres --command "CREATE USER foo WITH SUPERUSER ENCRYPTED PASSWORD 'bar';"
except this results in
psql: error: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
So what's the proper way to do this?
The Dockerfile is for building the image and the CMD will run when the container is ran. So the reason that
docker run --name $CONTAINER_NAME \
-p 5432:5432 \
-e POSTGRES_PASSWORD=mypass \
-e POSTGRES_DB=mydb \
-d postgres:13.5-alpine
###
### artificial wait here
###
docker exec $CONTAINER_NAME bash -c "psql --username postgres --command \"CREATE USER foo WITH SUPERUSER ENCRYPTED PASSWORD 'bar';\""
this worked is because the container (db) already is up and running before you input the SQL query into it. But if you try to do it from the Dockerfile it literally will try to run the psql command before the db is up and running, hence the connection error.
Your best bet is to use Docker Compose and add an initialize script
version: 3
services:
db:
image: postgres:13.5-alpine
environment:
POSTGRES_PASSWORD:mypass
POSTGRESS_DB: mydb
volume:
data:/var/lib/postgresql/data
volumes:
data:
Something like this(Excuse some typos. I'm typing from memory here.
See https://hub.docker.com/_/postgres for more info on initialising scripts
But it's basically a bash script you can write and then pass to docker compose to run upon container startup.
I'm trying to restore a .backup file to a Postgresql database. For that, I use a docker-compose file to launch a postgres docker container:
docker-compose.yml
postgresql:
image: postgres
restart: always
ports:
- "5432:5432"
environment:
- POSTGRES_USER:postgres
# - PGDATA:/var/lib/postgresql/data/pgdata/
volumes:
- ${PWD}/project/data/MX/bkup/data:/var/lib/postgresql
command:
- pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
When I run my docker-compose file using the command:
docker-compose up postgresql
I get the error:
(virtual) med#nid:~/projects/project/pkg$ docker-compose up postgresql
Recreating pkg_postgresql_1 ...
Recreating pkg_postgresql_1 ... done
Attaching to pkg_postgresql_1
postgresql_1 | /usr/local/bin/docker-entrypoint.sh: line 176: /pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup: No such file or directory
pkg_postgresql_1 exited with code 127
This happens even though the file is inside the volume
med#nid:~/projects/project/pkg$ docker-compose exec postgresql bash
root#ab7dbe2b0232:/# cd /var/lib/postgresql/
root#ab7dbe2b0232:/var/lib/postgresql# ls -l
total 1054780
drwx------ 19 postgres postgres 4096 Oct 2 08:51 data
-r--r--r-- 1 postgres ssl-cert 1080082803 Sep 27 15:40 ph.backup
I tried to use the -h argument of pg_restore in the docker-compose command:
pg_restore -h tcp://`docker-machine ip default`:5432 -U postgres -d postgres /var/lib/postgresql/ph.backup
What works:
If I comment the command target in the docker-compose.yml, launch the docker container and run the command inside it I get to have the data injected!
Is there a fix for this, Meaning, is there a way to make the command work directly from the docker-compose.yml file?
There are two forms of Docker Compose command:. You should move the command up on to the same line
command: pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
The form you have individually spells out each argument (in a YAML list); for example
command:
- /bin/ls
- -l
- -r
- -t
(Also consider just installing the PostgreSQL client tools on your host and running this outside of Docker; use localhost as the host name and the first number from the database container's ports: as the port number.)
The error comes from the form of the command configuration. If you inspect the finished postgres container (docker inspect <container-id>) the entrypoint and command look like this:
"Cmd": [
"pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup"
],
"Entrypoint": [
"docker-entrypoint.sh"
]
That practically means that the default entrypoint script docker-entrypoint
is executed with one argument which is the pg_restore command. At line 176 the script execs the passed in arguments (exec "$#").
The exec command needs a command and a list of arguments
exec [command [arguments]]
but in this case the command is the full string formed by pg_restore and its arguments. This obviously is not a valid file
Now, if you change command in docker-compose.yml to:
command: pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
inspecting the container shows the following:
"Cmd": [
"pg_restore",
"-U",
"postgres",
"-d",
"postgres",
"/var/lib/postgresql/ph.backup"
]
that means that exec will run pg_restore as command passing the rest as arguments and everything works as expected.
In alternative you could override the entrypoint in the docker-compose file to execute the command in a shell:
entrypoint: /bin/bash -c
command:
- pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
I am using the official postgresql docker image (version 9.4). I have extended the Dockerfile, so I can alter the settings in the postgresql.conf etc, using a bash script. It successfully adds and runs the script on entrypoint, for a single sed command. But when I put 2 or more sed commands, I get the following error:
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/config.sh
: No such file or directoryread
/var/lib/postgresql/data/postgresql.conf
I am trying on Windows 10, in combination with Vagrant and VirtualBox, using NFS file system on shared folders, via the vagrant-winnfsd plugin.
Why is this happening? How can I alter my bash script in order to work with more configuration settings? Is there a better way?
Dockerfile:
FROM postgres:9.4
RUN echo "Europe/Athens" > /etc/timezone \
&& dpkg-reconfigure -f noninteractive tzdata
RUN localedef -i el_GR -c -f UTF-8 -A /usr/share/locale/locale.alias el_GR.UTF-8
ADD config.sh /docker-entrypoint-initdb.d/
RUN chmod 755 /docker-entrypoint-initdb.d/config.sh
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
config.sh:
#!/bin/bash
sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
database.yml
postgres:
container_name: postgres-9.4
image: ***/postgres-9.4
volumes_from:
- postgres_data
ports:
- 5432:5432
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
- USERMAP_UID=999
- USERMAP_GID=999
postgres_data:
container_name: postgres_data
image: ***/postgres-9.4
volumes:
- ./services/postgres:/etc/postgresql
- ./services/postgres:/var/lib/postgresql
- ./services/postgres/logs:/var/log/postgresql
command: "true"
You might want to try using a RUN statement to execute your bash script or just run sed directly with both commands combined with a semicolon:
RUN sed -i -e 's/^#\(logging_collector = \).*/\1on/; s/^\(max_connections = \).*/\11000/' \
/var/lib/postgresql/data/postgresql.conf
A more scalable solution would be to put the sed program in an external file, then use these statements:
ADD postgres-edit.sed /var/local
RUN sed -i -f /var/local/postgres-edit.sed /var/lib/postgresql/data/postgresql.conf
postgres-edit.sed:
# sed script to edit postgresql configuration
s/^#\(logging_collector = \).*/\1on/
s/^\(max_connections = \).*/\11000/
Seems like a duplicate of How to customize the configuration file of the official PostgreSQL docker image?.
Copy-paste of my answer at https://stackoverflow.com/a/40598124/385548.
Inject custom postgresql.conf into postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize PostgreSQL configuration in Docker consistently, I suggest using config_file postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the postgres server with psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;
I using docker postgres:9.4 image.
I need to know how to backup and restore volume only container.
Created Volume container:
docker run -v /var/lib/postgresql/data --name dbdata postgres:9.4 /bin/true
Using the volume
docker run --name=postgres --volumes-from=dbdata -d -p 6432:5432 postgres:9.4
Backup Volume container
docker run --volumes-from dbdata -v $(pwd):/backup postgres:9.4 tar cvf /backup/backup.tar /var/lib/postgresql/data
Restore volume in new container
docker run --name=dbdata-new --volumes-from dbdata -v $(pwd):/backup ubuntu:14.04 /bin/sh -c 'cd /var/lib/postgresql/data && tar xvf /backup/backup.tar'
Use in the new volume in creating new postgres container:
docker run --name=postgres-new --volumes-from=dbdata-new -d -p 7532:5432 postgres:9.4
Issue: I get the below error in the logs when I run the new container.
initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data"
Not sure what I am doing wrong. Can someone please point out where I am making mistake.
Could not easily reproduce the issue following the steps with very rudiment data (one record one table one new db):
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -c "CREATE DATABASE ttt;"
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -d ttt -c "CREATE table a(b int); insert into a(b) values(1);"
psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 6432 -d ttt -c "select * from a;"
when I start the postgres-new I get no exceptions in logs and the data seems to be there:
$ psql -U postgres -h $(boot2docker ip || echo 'localhost') -p 7532 -d ttt -c "select * from a;"
b
---
1
(1 row)
data-new --volumes-from dbdata -v $(pwd):/backup ubuntu:14.04 /bin/sh -c 'cd /var/lib/postgresql/data && tar xvf /backup/backup.tar'
The bold text was the problem.
By untaring the backup in the folder /var/lib/postgresql/data ,the tar command was creating /var/lib/postgresql/data inside the above folder.
Thank you for all the help MyKola.