We have a web app using postgresql DB, being deployed to tomcat on CentOS7 env. We are using docker (and docker-compose), and running on an Azure visual machine.
We cannot pre-set the admin user 'postgres' password (e.g. mysecret) during the docker/docker-compoase build process.
We have tried to use the environments: setting from the docker-compose.yml file, also the ENV in the ./portgres Dockerfile file. Neither works.
I had to manually use 'docker exec -it /bin/bash' to run the psql command to set the password. We like to avoid the manual step.
$ cat docker-compose.yml
version: '0.2'
services:
app-web:
build: ./tomcat
ports:
- "80:80"
links:
- app-db
app-db:
build: ./postgres
environment:
- "POSTGRES_PASSWORD=password"
- "PGPASSWORD=password"
expose:
- "5432"
$ cat postgres/Dockerfile
FROM centos:7
RUN yum -y install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-centos96-9.6-3.noarch.rpm
RUN yum -y install postgresql96 postgresql96-server postgresql96-libs postgresql96-contrib postgresql96-devel
RUN yum -y install initscripts
USER postgres
ENV POSTGRES_PASSWORD=mysecret
ENV PGPASSWORD=mysecret
RUN /usr/pgsql-9.6/bin/initdb /var/lib/pgsql/9.6/data
RUN echo "listen_addresses = '*'" >> /var/lib/pgsql/9.6/data/postgresql.conf
RUN echo "PORT = 5432" >> /var/lib/pgsql/9.6/data/postgresql.conf
RUN echo "local all all trust" > /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all 127.0.0.1/32 trust" >> /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all ::1/128 ident" >> /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/pgsql/9.6/data/pg_hba.conf
EXPOSE 5432
ENTRYPOINT ["/usr/pgsql-9.6/bin/postgres","-D","/var/lib/pgsql/9.6/data","-p","5432"]
Web app deployment fails, db authentication error with wrong password (the password 'mysecret' is defined in the web app JPA persistence.xml). Assuming the password was not properly set (default initdb does not set the password)
Then, manually change the password using the above mentioned docker exec command, everything works.
Like to set the password during the docker build time. Based on Postgres/Docker documentation and some threads, either environments: from docker-compose setting or ENV from the docker file would work. Neither works for us.
Related
I am getting port issue in docker compose.
My compose file is:
version: '3.1'
services:
db:
hostname: postgres
build:
context: .
dockerfile: Dockerfile.postgres
restart: always
ports:
- "5432:5432"
test:
build:
context: ../..
dockerfile: Dockerfile.test
environment:
DB_URL: "jdbc:postgresql://postgres:5432/postgres?user=postgres&password=postgres"
depends_on:
- postgres
My docker file for test is
FROM openjdk:8
RUN \
curl -L -o sbt-1.2.8.deb http://dl.bintray.com/sbt/debian/sbt-1.2.8.deb && \
dpkg -i sbt-1.2.8.deb && \
rm sbt-1.2.8.deb && \
apt-get update && \
apt-get install sbt && \
sbt sbtVersion
ENV WORK_DIR="/test"
USER root
COPY . ${WORK_DIR}/
RUN cd ${WORK_DIR} && \
sbt test
While running it i am getting:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
TEST_DB_URL is an env variable in application.conf.
I am running the individual commands:
docker compose -f docker-compose.yml up db
It works fine and docker ps show
0.0.0.0:5432->5432/tcp
But when i run docker compose -f docker-compose.yml up test i downloads dependencies and compiles the code and give me Connection to localhost:5432 refused.
Looks like this has nothing to do with connecting to the database, but is a configuration issue. It's using the default from the application.conf instead of using the override from the environment. You could try to narrow it down by simply printing the config in Scala and figuring out why the ConfigFactory doesn't pick up the environment variable.
Gitlab CI keeps ignoring the sql-files in /docker-entrypoint-initdb.d/* in this project.
here is docker-compose.yml:
version: '3.6'
services:
testdb:
image: postgres:11
container_name: lbsn-testdb
restart: always
ports:
- "65432:5432"
volumes:
- ./testdb/init:/docker-entrypoint-initdb.d
here is .gitlab-ci.yml:
stages:
- deploy
deploy:
stage: deploy
image: debian:stable-slim
script:
- bash ./deploy.sh
The deployment script basically uses rsync to deploy the content of the repository to to the server via SSH:
rsync -rav --chmod=Du+rwx,Dgo-rwx,u+rw,go-rw -e "ssh -l gitlab-ci" --exclude=".git" --delete ./ "gitlab-ci#$DEPLOY_SERVER:test/"
and then ssh's into the server to stop and restart the container:
ssh "gitlab-ci#$DEPLOY_SERVER" "cd test && docker-compose down && docker-compose up --build --detach"
This all goes well, but when the container starts up, it is supposed to run all the files that are in /docker-entrypoint-initdb.d/* as we can see here.
But instead, when doing docker logs -f lbsn-testdb on the server, I can see it stating
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
and I have no clue, why that happens. When running this container locally or even when I ssh to that server, clone the repo and bring up the containers manually, it all goes well and parses the sql-files. Just not when the Gitlab CI does it.
Any ideas on why that is?
This has been easier than I expected, and fatally nothing to do with Gitlab CI but with file permissions.
I passed --chmod=Du+rwx,Dgo-rwx,u+rw,go-rw to rsync which looked really secure because only the user can do stuff. I confess that I propably copypasted it from somewhere on the internet. But then the files are mounted to the Docker container, and in there they have those permissions as well:
-rw------- 1 1005 1004 314 May 8 15:48 100-create-database.sql
On the host my gitlab-ci user owns those files, they are obviously also owned by some user with ID 1005 in the container as well, and no permissions are given to other users than this one.
Inside the container the user who does things is postgres though, but it can't read those files. Instead of complaining about that, it just ignores them. That might be something to create an issue about…
Now that I pass --chmod=D755,F644 it looks like that:
-rw-r--r-- 1 1005 1004 314 May 8 15:48 100-create-database.sql
and the docker logs say
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/100-create-database.sql
Too easy to think of in the first place :-/
If you already run the postgres service before, the init files will be ignored when you restart it so try to use --build to build the image again
docker-compose up --build -d
and before you run again :
Check the existing volumes with
docker volume ls
Then remove the one that you are using for you pg service with
docker volume rm {volume_name}
-> Make sure that the volume is not used by a container, if so then remove the container as well
I found this topic discovering a similar problem with PostgreSQL installation using the docker-compose tool.
The solution is basically the same. For the provided configuration:
version: '3.6'
services:
testdb:
image: postgres:11
container_name: lbsn-testdb
restart: always
ports:
- "65432:5432"
volumes:
- ./testdb/init:/docker-entrypoint-initdb.d
Your deployment script should set 0755 permissions to your postgres container volume, like chmod -R 0755 ./testdb in this case. It is important to make all subdirectories visible, so chmod -R option is required.
Official Postgres image is running under internal postgres user with the UID 70. Your application user in the host is most likely has different UID like 1000 or something similar. That is the reason for postgres init script to miss installation steps due to permissions error. This issue appears several years, but still exist in the latest PostgreSQL version (currently is 12.1)
Please be aware of security vulnerability when having readable for all init files in the system. It is good to use shell environment variables to pass secrets into the init scrip.
Here is a docker-compose example:
postgres:
image: postgres:12.1-alpine
container_name: app-postgres
environment:
- POSTGRES_USER
- POSTGRES_PASSWORD
- APP_POSTGRES_DB
- APP_POSTGRES_SCHEMA
- APP_POSTGRES_USER
- APP_POSTGRES_PASSWORD
ports:
- '5432:5432'
volumes:
- $HOME/app/conf/postgres:/docker-entrypoint-initdb.d
- $HOME/data/postgres:/var/lib/postgresql/data
Corresponding script create-users.sh for creating users may looks like:
#!/bin/bash
set -o nounset
set -o errexit
set -o pipefail
POSTGRES_USER="${POSTGRES_USER:-postgres}"
POSTGRES_PASSWORD="${POSTGRES_PASSWORD}"
APP_POSTGRES_DB="${APP_POSTGRES_DB:-app}"
APP_POSTGRES_SCHEMA="${APP_POSTGRES_SCHEMA:-app}"
APP_POSTGRES_USER="${APP_POSTGRES_USER:-appuser}"
APP_POSTGRES_PASSWORD="${APP_POSTGRES_PASSWORD:-app}"
DATABASE="${APP_POSTGRES_DB}"
# Create single database.
psql --variable ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --command "CREATE DATABASE ${DATABASE}"
# Create app user.
psql --variable ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --command "CREATE USER ${APP_POSTGRES_USER} SUPERUSER PASSWORD '${APP_POSTGRES_PASSWORD}'"
psql --variable ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --command "GRANT ALL PRIVILEGES ON DATABASE ${DATABASE} TO ${APP_POSTGRES_USER}"
psql --variable ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${DATABASE}" --command "CREATE SCHEMA ${APP_POSTGRES_SCHEMA} AUTHORIZATION ${APP_POSTGRES_USER}"
psql --variable ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --command "ALTER USER ${APP_POSTGRES_USER} SET search_path = ${APP_POSTGRES_SCHEMA},public"
this is a circleci question I guess.
I am quite happy with circleci but now I ran into a problem and I don't know what I'm doing wrong.
Maybe this is something very easy, but I don't see the it.
In short
I can't make containers talk to each other on circleci.
Problem
Basically what I wanted to do is start a server container and a client container, and then let them talk to each other.
I created a minimal example here: https://github.com/mRcSchwering/circleci-integration-test
The README.md basically explains the desired outcome.
I have a .circleci/config.yml like this:
version: 2
jobs:
build:
docker:
- image: docker:18.03.0-ce-git
steps:
- checkout
- setup_remote_docker
- run:
name: Install docker-compose
command: |
apk --update add py2-pip
/usr/bin/pip2 install docker-compose
docker-compose --version
- run:
name: Start Container
command: |
docker-compose up -d
docker-compose ps
- run:
name: Let client talk to server
command: |
docker-compose run client psql -h server -p 5432 -U postgres -c "\l"
In a docker container, docker-compose is installed, which is then used to start a server and a client (postgres here). In the last step I am telling the client to query the server. However, it cannot find the server:
#!/bin/sh -eo pipefail
docker-compose run client psql -h server -p 5432 -U postgres -c "\l"
Starting project_server_1 ...
^#^#psql: could not connect to server: Connection refused
Is the server running on host "server" (172.18.0.2) and accepting
TCP/IP connections on port 5432?
Exited with code 2
Files
The docker-compose.yml looks like this
version: '2'
services:
server:
image: postgres:9.5.12-alpine
networks:
- internal
expose:
- '5432'
client:
build:
context: .
networks:
- internal
depends_on:
- server
networks:
internal:
driver: bridge
where the client is built from a dockerfile like this
FROM alpine:3.7
RUN apk --no-cache add postgresql-client && rm -rf /var/cache/apk/*
Note
If I repeat everything on my Linux (also with docker-in-docker) it works.
But I guess some things work completely different on circleci.
I found some people mentioning that on circleci networking and bind mounts can be tricky but I didn't find anything that can help me.
There is this doc but I thought I am doing this already.
Then there is this project where someone seems to do the same thing on circleci successfully.
But I cannot figure out what's different there...
Anyway I would really appreciate your help. So far I have given up on this.
Best
Marc
Ok, in the meanwhile I (no actually it was halfer from the circleci forum) noticed that docker-compose run client psql -h server -p 5432 -U postgres -c "\l" was run before the server was up and running. A simple sleep 5 after docker-compose up -d fixes the problem.
I'm pretty new to docker-compose and I'm trying to build a container with postgres 9.6 on top of centos 6 and running it with docker compose up.
Then I want to be able to connect to the database through some db management tool.
If I run:
docker-compose up
The image builds correctly but then I can't connect to the database.
If I remove the tty:true and stdin_open: true the image builds but then it exits with exit code 0
If I run docker-run -it -p 5432:5432 --entypoint /bin/bash [image_name] and then i run manually the command service postgresql-9.6 start everything works fine (until I exit the container).
Any suggestion?
Here's my docker-compose file:
networks{}
version: '2'
services:
postgres:
build:
context: ./src/test/docker/postgres
ports:
- "5432:5432"
stdin_open: true
tty: true
And my docker file:
FROM centos:6
RUN yum -y install
https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-
centos96-9.6-3.noarch.rpm
RUN yum -y install postgresql96 postgresql96-server postgresql96-libs
postgresql96-contrib postgresql96-devel
# Initialize the database (not starting it yet)
RUN service postgresql-9.6 initdb
RUN su postgres
RUN echo "listen_addresses = '*'" >> /var/lib/pgsql/9.6/data/postgresql.conf
RUN echo "PORT = 5432" >> /var/lib/pgsql/9.6/data/postgresql.conf
RUN echo "local all all trust" > /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all 127.0.0.1/32 ident" >> /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all ::1/128 ident" >> /var/lib/pgsql/9.6/data/pg_hba.conf
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/pgsql/9.6/data/pg_hba.conf
RUN exit
# Expose the PostgreSQL port
EXPOSE 5432
ENTRYPOINT service postgresql-9.6 start
An ENTRYPOINT in your docker file allows you to configure your container as an executable. It's a way of wrapping what you need run with all dependencies in a container and running the container = running your script/application etc. See documentation details here
Now your entry point
ENTRYPOINT service postgresql-9.6 start
Only runs the service start and exits! So your container will also exit.
If you want to configure it like an executable you need to run the postgres command as your ENTRYPOINT command
ENTRYPOINT ["sudo","-u","postgres","/usr/pgsql-9.6/bin/postgres","-D","/var/lib/pgsql/9.6/data","-p","5432"]
This will run postgres as your ENTRYPOINT command whenever the container starts, and you should be able to connect to it.
I'm running postgres inside a docker container to limit the amount of system resources it has access to. I'm having some trouble understanding how to make the data persistent. I've read the following articles:
https://www.andreagrandi.it/2015/02/21/how-to-create-a-docker-image-for-postgresql-and-persist-data/
http://container42.com/2013/12/16/persistent-volumes-with-docker-container-as-volume-pattern/
Which suggest using a data only container, and then having my postgres container link to it. What I'm failing to understand is; what's the advantage to this? As far as I can tell, if for some reason the docker-machine shut down (for example; moving it to a different physical machine), the data only container stops running, and all of it's contents are lost? I've tried creating a volume in the postgres container, but it doesn't actually seem to save anything to the disk.
Here's my docker file. What am I doing wrong?
FROM ubuntu
MAINTAINER Andrew Broadbent <andrew.broadbent#manchester.ac.uk>
# Add the PostgreSQL PGP key to verify their Debian packages.
# It should be the same key as https://www.postgresql.org/media/keys/ACCC4CF8.asc
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
# Add PostgreSQL's repository. It contains the most recent stable release
# of PostgreSQL, ``9.3``.
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list
# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
# There are some warnings (in red) that show up during the build. You can hide
# them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
# after each ``apt-get``
# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres
# Create a PostgreSQL role named ``docker`` with ``docker`` as the password and
# then create a database `docker` owned by the ``docker`` role.
# Note: here we use ``&&\`` to run commands one after the other - the ``\``
# allows the RUN command to span multiple lines.
RUN /etc/init.d/postgresql start &&\
psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
createdb -O docker docker
# Complete configuration
USER root
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
# Expose the PostgreSQL port
EXPOSE 5432
# Add VOLUMEs to allow backup of config, logs and databases
RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
# Set the default command to run when starting the container
USER postgres
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
This one answers your question about data container:
docker mounting volumes on host
Regarding to your dockerfile, I would suggest you either:
1) use data container pattern
2) mount the volume to host machine by specifying: docker run -v [host-path]:[container-path] ..., so that data will be kept at one place in your host and will not be lost after the container is removed.
Ref: https://docs.docker.com/engine/tutorials/dockervolumes/#/mount-a-host-directory-as-a-data-volume