How to run the python application and postgres in one docker container? - postgresql

I have a python application that interacts with the postgresql database and i need to run it all in one docker container.
I get a connection error when running the container:
...
File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/usr/lib/python3.6/asyncio/base_events.py", line 787, in create_connection
', '.join(str(exc) for exc in exceptions)))
OSError: Multiple exceptions: [Errno 111] Connect call failed ('::1', 5432), [Errno 111] Connect call failed ('127.0.0.1', 5432)
My Dockerfile:
FROM postgres:10.0-alpine
RUN apk add --update --no-cache g++ alpine-sdk
RUN apk --no-cache add python3-dev
RUN apk add --no-cache python3 && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache
ADD app /app/
RUN chmod -R 777 /app
WORKDIR /app
ADD requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
USER postgres
RUN chmod 0700 /var/lib/postgresql/data &&\
initdb /var/lib/postgresql/data &&\
echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf &&\
echo "listen_addresses='*'" >> /var/lib/postgresql/data/postgresql.conf &&\
pg_ctl start &&\
psql --command "ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';"
EXPOSE 5432
EXPOSE 80
CMD ["python3", "main.py"]

Although this is not recommended, it's doable. The problem is pg_ctl in RUN instruction is executed at build time, not in the container. You need to run it with CMD.
You can have a script like
pg_ctl start
psql --command "ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';"
python3 main.py
COPY the script in the image and at the end of the dockerfile, `CMD ["./script.sh"]

#Anton, it is not recommended to run multiple processes inside a docker container. Have a look at this link https://docs.docker.com/config/containers/multi-service_container/ it will explain more and demonstrate a manner to accomplish this. You probably are aware, but your container will be running the postgresql instance and have the data in it, so if you ever re-create the container, you will lose any data in that container.

Related

Postgres Docker container failed to start with psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" f

I am working on a docker container which initialises postgre sql db from a sql dump file.However while running the init script I am facing below exception .
Attaching to postgres
postgres | psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
postgres | Is the server running locally and accepting connections on that socket?
postgres exited with code 2
DockerFile:
FROM postgres:latest
## Some utilities
RUN apt-get update -y && \
apt-get install -y build-essential libfuse-dev libcurl4-openssl-dev libxml2-dev pkg-config libssl-dev mime-support automake libtool wget tar git unzip
RUN apt-get install lsb-release -y && apt-get install zip -y && apt-get install vim -y
## Install S3 Fuse
RUN rm -rf /usr/src/s3fs-fuse
RUN git clone https://github.com/s3fs-fuse/s3fs-fuse/ /usr/src/s3fs-fuse
WORKDIR /usr/src/s3fs-fuse
RUN ./autogen.sh && ./configure && make && make install
## Create folder
RUN mkdir /data
WORKDIR /data
RUN mkdir out
## change workdir to /
WORKDIR /
RUN rm -rf 1-restore.sh
ADD docker-entrypoint-initdb.d/1-restore.sh 1-restore.sh
RUN chmod 777 1-restore.sh
ENTRYPOINT ["/bin/bash", "1-restore.sh" ]
Below is my init Shell script file:
#!/bin/bash
set -e
export PGPASSWORD=$POSTGRES_PASSWORD
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < $DUMP_FILE
set -e
export PGPASSWORD=$POSTGRES_PASSWORD
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
SET client_encoding TO 'UTF8';
\t
\a
\o $OUT_FOLDER/export_json.sql;
SELECT '\o $OUT_FOLDER/' || table_name || '.json;' || E'\n' || 'SELECT row_to_json(r) FROM "' || table_name || '" AS r;'
FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY table_name;
\i $OUT_FOLDER/export_json.sql;
EOSQL
rm $OUT_FOLDER/export_json.sql
echo ""TestMessage">>test.txt
I checked the other so ans but nothing yet worked for me .

How to prevent docker from exiting after postgres start

I'm trying to build a customized postgres replication cluster in docker. I'm using postgres:9.6-alpine as my base image.
Here is my Dockerfiles which I'm using to build the slave container.
// Dockerfile
FROM postgres:9.6-alpine
ENV GOSU_VERSION 1.10
RUN set -ex; \
\
apk add --no-cache --virtual .gosu-deps \
dpkg \
gnupg \
openssl \
; \
\
dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')"; \
wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch"; \
wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc"; \
\
# verify the signature
export GNUPGHOME="$(mktemp -d)"; \
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \
gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc; \
\
chmod +x /usr/local/bin/gosu; \
# verify that the binary works
gosu nobody true; \
\
apk del .gosu-deps
RUN apk add --update iputils
RUN apk add --update htop
# COPY ./setup-slave.sh /docker-entrypoint-initdb.d
COPY ./docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh
CMD ["gosu", "postgres", "pg_ctl", "-D/var/lib/postgresql/data", "start"]
And this is my docker-entrypoint.sh:
#!/bin/bash
if [ ! -s "$PGDATA/PG_VERSION" ]; then
echo '*:*:*:myuser:123456' > ~/.pgpass
chmod 0600 ~/.pgpass
until ping -c 1 -W 1 pg_master
do
echo "Waiting for master to ping..."
sleep 1s
done
until pg_basebackup -h pg_master -D ${PGDATA} -U arioo -vP -W
do
echo "Waiting for master to connect..."
sleep 1s
done
echo "host replication all 0.0.0.0/0 md5" >> "$PGDATA/pg_hba.conf"
set -e
cat >> ${PGDATA}/postgresql.conf <<EOF
wal_level = hot_standby
max_wal_senders = 8
wal_keep_segments = 32
hot_standby = off
EOF
cat > ${PGDATA}/recovery.conf <<EOF
standby_mode = on
primary_conninfo = 'host=pg_master port=5432 user=rep password=123456'
trigger_file = '/tmp/touch_me_to_promote_to_me_master'
EOF
chown postgres. ${PGDATA} -R
chmod 700 ${PGDATA} -R
fi
exec "$#"
The whole thing works just fine except for one thing. Docker immediately exits the slave container upon start. From my understanding pg_ctl start should start the postgres in foreground. But why is docker exiting still?
I use docker-compose to run the cluster.
I even tried putting an endless loop at end of the docker-entrypoint.sh like the code below and it's still exiting no matter what I do:
while true; do
sleep 1s
done
It seems that running postgres instead of pg_ctl start was the solution. pg_ctl is actually a utility to control the postgres daemon so it is logical that it runs postgres in the background.

Enable Postgis extension Postgres in Dockerfile

How can i enable the Postgis extension in a Dockerfile? The Postgres installation is already working
this is the command in Ubuntu:
sudo -u postgres psql -c "CREATE EXTENSION postgis;"
My Dockerfile:
# Set the base image to Ubuntu
FROM ubuntu:14.04
# Update the repository sources list
RUN apt-get update -y
################## BEGIN INSTALLATION ######################
# Postgres with Postgis
# Install wget
RUN apt-get install wget -y
# Setup Postgres repository
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Add Postgres repository
RUN sh -c "echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" >> /etc/apt/sources.list.d/postgresql.list"
# Update repository
RUN apt-get update -y
# Install Postgres with Postgis
RUN apt-get install postgresql-9.3-postgis-2.1 -y
# Change rights for start Postgresql
RUN chmod +x /etc/init.d/postgresql
# Start Postgresql
CMD service postgresql start && tail -F /var/lib/postgresql/data/serverlog
The solution for enable an extension
# Enable Postgis
RUN service postgresql start \
&& sudo -u postgres psql -c "CREATE EXTENSION postgis;"
RUN service postgresql stop
If you see other Dockerfiles using that psql -c command, like apache/marmotta Dockerfile, you will see lines like:
RUN service postgresql start \
&& psql --command "CREATE USER $DB_USER WITH PASSWORD '$DB_PASS';" \
&& psql --command "CREATE DATABASE $DB_NAME WITH OWNER $DB_USER;"
USER root
RUN service postgresql stop
In other words, you need to make sure the postgresql service is running for those commands to succeed.

Codenvy add postgresql to the project error

I am new at Codenvy so my question can be very stupid, by the way this is my problem:
I created my project then to add the DB (postgresql) i created a new docker file and i pasted in it the code in this file: https://github.com/codenvy/dockerfiles/blob/master/base/jdk7_postgresql/Dockerfile
I saved it then i ran the project with the runner just created, but i have this error:
[DOCKER] Setting up dh-python (1.20141111-2) ...
[DOCKER] Processing triggers for systemd (215-17+deb8u2) ...
[DOCKER] Processing triggers for libc-bin (2.19-18+deb8u1) ...
[DOCKER] Processing triggers for dbus (1.8.20-0+deb8u1) ...
[DOCKER] Starting PostgreSQL 9.3 database server:
[DOCKER] main
[DOCKER] .
[DOCKER] CREATE ROLE
[DOCKER] ---> 77b708d3360b
[DOCKER] Removing intermediate container cdd908fb498e
[DOCKER] Step 4 : ADD startup.sh /home/user/startup.sh
[DOCKER][ERROR] startup.sh: no such file or directory
[ERROR] We are having trouble starting the runner and deploying application. Either necessary files are missing or a fundamental configuration has changed.
Docker image build failed
Any idea to how can i solve the problem?
thanks in advance!
This PostgresSQL image uses startup.sh script to start postgresql service. You can find it at https://github.com/codenvy/dockerfiles/blob/master/base/jdk7_postgresql/startup.sh
To solve this problem you can either create a new file startup.sh with an identical content in the root of your project, add it as a source (e.g. ADD $src$/startup.sh /home/user/startup.sh). You will also need to inject project sources to make it work as expected, so as a result your Dockerfile may look like this:
FROM codenvy/jdk7
ENV DEBIAN_FRONTEND noninteractive
RUN sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8 && \
echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list && \
sudo apt-get update && \
sudo -E bash -c "apt-get install -y python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3 pwgen" && \
sudo service postgresql start && \
CODENVY_POSTGRESQL_PASSWORD=$(pwgen -N 1) && echo "export CODENVY_POSTGRESQL_PASSWORD=$CODENVY_POSTGRESQL_PASSWORD" >> /home/user/.postgresrc && \
CODENVY_POSTGRESQL_DB=testdb_$(pwgen -N 1) && echo "export CODENVY_POSTGRESQL_DB=$CODENVY_POSTGRESQL_DB" >> /home/user/.postgresrc && \
CODENVY_POSTGRESQL_USER=codenvy && echo "export CODENVY_POSTGRESQL_USER=$CODENVY_POSTGRESQL_USER" >> /home/user/.postgresrc && \
sudo -u postgres psql --command "CREATE USER $CODENVY_POSTGRESQL_USER WITH SUPERUSER PASSWORD '$CODENVY_POSTGRESQL_PASSWORD';" && \
sudo -u postgres createdb -O $CODENVY_POSTGRESQL_USER $CODENVY_POSTGRESQL_DB
#Inject project sources
ADD $app$ /home/user/$app$
RUN unzip -q /home/user/$app$ -d /home/user
ENV ARGUMENTS $args$
#Add file to the container
ADD $src$/startup.sh /home/user/startup.sh
RUN sudo chmod +x /home/user/startup.sh
#EXPOSE 5432
CMD sudo /home/user/startup.sh
Or you can simply use Codenvy default PostgreSQL 9.3 + Java 7 runner, which you will find on the Runners panel > Configs tab.

How to create docker image for postgis that will enable extension at build time or before container fully running?

What I mean is that I want to create a docker image for postgis that will be completely usable right after build. So that if user runs
docker run -e POSTGRES_USER=user somepostgis
the user database would be created and extensions already installed?
The official postgres image can't be used for that AFAIK.
Basically need to write script and tell that it would be entrypoint. This script should create database and create extensions with porstgres server running on different port and then restart it on port 5432.
But I don't know sh enough and docker to do that. Right now it's saying that there is no pg_ctl command
If you want to help you can fork
FROM ubuntu:15.04
#ENV RELEASE_NAME lsb_release -sc
#RUN apt-get update && apt-get install wget
#RUN echo "deb http://apt.postgresql.org/pub/repos/apt ${RELEASE_NAME}-pgdg main" >> /etc/apt/sources.list
#RUN cat /etc/apt/sources.list
#RUN wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-9.4-postgis-2.1 \
curl \
&& curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture)" \
&& curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture).asc" \
&& gpg --verify /usr/local/bin/gosu.asc \
&& rm /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& apt-get purge -y --auto-remove curl
RUN mkdir /docker-entrypoint-initdb.d
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
RUN chmod +x /docker-entrypoint.sh
RUN ls -l /docker-entrypoint.sh
EXPOSE 5432
CMD ["postgres"]
So I'm trying to do somethink like that, but it doesn't work.
#!/bin/bash
${POSTGRES_DB:=$POSTGRES_USER}
gosu postgres pg_ctl start -w -D ${PGDATA} -0 "-p 5433"
gosu postgres createuser ${POSTGRES_USER}
gosu postgres createdb ${POSTGRES_DB} -s -E UTF8
gosu postgres psql -d ${POSTGRES_DB} -c "create extension if not exists postgis;"
gosu postgres psql -d ${POSTGRES_DB} -c "create extension if not exists postgis_topology;"
pg_ctl -w restart