Why can't I run pdo_pgsql in my container? - postgresql

I am trying to use AWS RDS postgresql from my php app. I have modified the Dockerfile to contain
RUN apt-get update && apt-get install -y libpq-dev && docker-php-ext-install pdo pdo_pgsql
But it doesn't seem to run. I think this because when I (finally) ran phpinfo() from the container, I don't see it loaded. I am assuming that when I run docker-compose up, the Dockerfile (that builds php-fpm) is run "automatically". Is that true? Here is the Dockerfile that doesn't seem to run:
FROM bitnami/php-fpm:7.2 as builder
RUN install_packages git autoconf build-essential
WORKDIR /app
RUN wget https://github.com/xdebug/xdebug/archive/2.6.0.tar.gz && \
tar xzf 2.6.0.tar.gz && \
cd xdebug-2.6.0 && \
phpize && \
./configure --enable-xdebug && \
make && make install
RUN apt-get update && apt-get install -y libpq-dev && docker-php-ext-install pdo pdo_pgsql
FROM bitnami/php-fpm:7.2
COPY --from=builder /opt/bitnami/php/lib/php/extensions/xdebug.so /opt/bitnami/php/lib/php/extensions/
RUN echo 'zend_extension="/opt/bitnami/php/lib/php/extensions/xdebug.so"' >> /opt/bitnami/php/etc/php.ini
RUN echo "xdebug.remote_port=9000" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.remote_enable=1" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.remote_connect_back=0" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.remote_host=192.168.122.1" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.idekey=docker" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.remote_autostart=1" >> /opt/bitnami/php/etc/php.ini \
&& echo "xdebug.remote_log=/tmp/xdebug.log" >> /opt/bitnami/php/etc/php.ini \
&& echo "extension=pgp_pdo_pgsql.so" >> /opt/bitnami/php/etc/php.ini \
&& echo "extension=pgp_pgsql.so" >> /opt/bitnami/php/etc/php.ini
And here is the docker-compose
version: '3'
services:
apache:
image: bitnami/apache:latest
restart: unless-stopped
ports:
- 80:8080
volumes:
- ./apache/app.conf:/vhosts/app.conf:ro
- ./app:/app
networks:
- net
mysql:
container_name: "mysql"
restart: unless-stopped
image: mysql:5.6
environment:
- MYSQL_DATABASE
- MYSQL_PASSWORD
- MYSQL_ROOT_PASSWORD
- MYSQL_USER
volumes:
- data:/var/lib/mysql
- ./mysql/mysql.sql:/docker-entrypoint-initdb.d/mysql.sql
ports:
- "3306:3306"
networks:
- net
phpmyadmin:
container_name: phpmyadmin
restart: unless-stopped
image: phpmyadmin/phpmyadmin
environment:
- PMA_HOST=mysql
- PMA_PORT=3306
ports:
- "8081:80"
networks:
- net
php-fpm:
build: ./php
restart: unless-stopped
image: bitnami/php-fpm
volumes:
- ./app:/app
environment:
- XDEBUG_CONFIG="remote_host=192.168.122.1"
networks:
- net
jasperreports:
image: 'bitnami/jasperreports:7'
restart: unless-stopped
environment:
- MARIADB_HOST=mysql
- MARIADB_PORT_NUMBER=3306
- JASPERREPORTS_USERNAME=admin
- JASPERREPORTS_PASSWORD=bitnami
- JASPERREPORTS_DATABASE_USER=admin
- JASPERREPORTS_DATABASE_PASSWORD=xxx
- JASPERREPORTS_DATABASE_NAME=jasper
- ALLOW_EMPTY_PASSWORD=yes
ports:
- '8080:8080'
volumes:
- jasperreports_data:/bitnami
depends_on:
- mysql
networks:
- net
volumes:
data:
driver: local
jasperreports_data:
driver: local
If this Dockerfile is not being run automatically, how do I get it to create a new php-fpm container?

If you change Dockerfile and you want to update the image with docker-compose up, then you need to pass the flag --build with docker-compose up.
--build Build images before starting containers.
docker-compose up --build
Ref:- https://docs.docker.com/compose/reference/up/

Related

pg_dump from Celery container differs from pg_dump in other containers

I can't understand where pg_dump version is coming from.
I forced everywhere postrgresql-client-13 to be installed.
/usr/bin/pg_dump --version
Celery Beat and Celery
pg_dump (PostgreSQL) 11.12 (Debian 11.12-0+deb10u1)
Other containers (web & postgre and local machine) :
pg_dump (PostgreSQL) 13.4 (Debian 13.4-1.pgdg100+1)
Here is my Dockerfile
FROM python:3
#testdriven turotial they use an other user than root but seemed to fail ehre .
# create directory for the app user
RUN mkdir -p /home/app
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/staticfiles
WORKDIR $APP_HOME
ENV PYTHONUNBUFFERED 1
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
RUN apt-get update -qq && apt-get install -y \
postgresql-client-13 \
binutils \
libproj-dev \
gdal-bin
RUN apt-get update \
&& apt-get install -yyq netcat
# install psycopg2 dependencies
#install dependencies
RUN pip3 install --no-cache-dir --upgrade pip && pip install --no-cache-dir --no-cache-dir -U pip wheel setuptools
COPY ./requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# copy entrypoint-prod.sh
COPY ./entrypoint.prod.sh $APP_HOME
# copy project
COPY . $APP_HOME
# run entrypoint.prod.sh
ENTRYPOINT ["/home/app/web/entrypoint.prod.sh"]
and here is my docker-compose
version: '3.7'
services:
web:
build:
context: ./app
dockerfile: Dockerfile.prod
command: gunicorn core.wsgi:application --bind 0.0.0.0:8000
volumes:
- static_volume:/home/app/web/staticfiles
expose:
- 8000
env_file:
- ./app/.env.prod
depends_on:
- db
db:
image: postgis/postgis:13-master
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- ./app/.env.prod.db
redis:
image: redis:6
celery:
build: ./app
command: celery -A core worker -l info
volumes:
- ./app/:/usr/src/app/
env_file:
- ./app/.env.prod
depends_on:
- redis
celery-beat:
build: ./app
command: celery -A core beat -l info
volumes:
- ./app/:/usr/src/app/
env_file:
- ./app/.env.prod
depends_on:
- redis
nginx-proxy:
image : nginxproxy/nginx-proxy:latest
container_name: nginx-proxy
build: nginx
restart: always
ports:
- 443:443
- 80:80
volumes:
- static_volume:/home/app/web/staticfiles
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- /var/run/docker.sock:/tmp/docker.sock:ro
depends_on:
- web
nginx-proxy-letsencrypt:
image: jrcs/letsencrypt-nginx-proxy-companion
env_file:
- ./app/.env.prod.proxy-companion
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
depends_on:
- nginx-proxy
volumes:
postgres_data:
static_volume:
certs:
html:
vhost:
I really need to have Celery with the same pg_dump version.
Can you guys provide some inputs ?

run script on docker file to wait postgress

Hello i'm new to docker and linux and i don't know how to run a script for my docker to run after my postgress
this is my script:
until psql -c '\l'; do
echo >&2 "$(date +%Y%m%dt%H%M%S) Postgres is unavailable - sleeping"
sleep 1
done
echo >&2 "$(date +%Y%m%dt%H%M%S) Postgres is up - executing command"
exec ${#}
my docker file :
FROM node:lts-alpine
RUN mkdir -p /home/node/api/node_modules && chown -R node:node /home/node/api
WORKDIR /home/node/api
COPY wait-pg.sh ormconfig.json .env package.json yarn.* ./
USER node
RUN yarn
COPY --chown=node:node . .
EXPOSE 4000
RUN chmod +x /wait-pg.sh
CMD ["yarn", "dev"]
my docker compose:
version: '3.7'
services:
db-pg:
image: postgres:12
container_name: db-pg
ports:
- '${DB_PORT}:5432'
environment:
ALLOW_EMPTY_PASSWORD: 'no'
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASS}
POSTGRES_DB: ${DB_NAME}
volumes:
- ci-postgres-data:/data
ci-api:
build: .
container_name: ci-api
volumes:
- .:/home/node/api
- /home/node/api/node_modules
ports:
- '${SERVER_PORT}:${SERVER_PORT}'
depends_on:
- db-pg
logging:
driver: 'json-file'
options:
max-size: '10m'
max-file: '5'
volumes:
ci-postgres-data:
If someone can help me like I would in my docker compose and in my dockerfile.
I would like to know how I can add my script to my docker file and docker compose
Check the below Dockerfile
FROM node:lts-alpine
RUN mkdir -p /home/node/api/node_modules && chown -R node:node /home/node/api
WORKDIR /home/node/api
COPY wait-pg.sh ormconfig.json .env package.json yarn.* ./
USER node
RUN yarn
COPY --chown=node:node . .
EXPOSE 4000
CMD ["yarn", "dev"]
Please try the below docker-compose.yml in which depends_on, healthcheck and links are added as web service depends on db service.
version: "2.1"
services:
ci-api:
build: .
container_name: ci-api
volumes:
- .:/home/node/api
- /home/node/api/node_modules
ports:
- '${SERVER_PORT}:${SERVER_PORT}'
links:
- db-pg
depends_on:
- db-pg
logging:
driver: 'json-file'
options:
max-size: '10m'
max-file: '5'
db-pg:
image: postgres:12
container_name: db-pg
ports:
- '${DB_PORT}:5432'
environment:
ALLOW_EMPTY_PASSWORD: 'no'
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASS}
POSTGRES_DB: ${DB_NAME}
volumes:
- ci-postgres-data:/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
interval: 10s
timeout: 5s
retries: 5
volumes:
ci-postgres-data:

Installing and using pg_cron extension on Postgres running inside of Docker container

I tried installing pg_cron on Postgres running inside a Docker container but getting this error could not access file "pg_cron": No such file or directory. Any ideas on how to resolve?
Based on https://stackoverflow.com/a/51797554, I tried the following:
docker-compose.yml
version: '3.7'
services:
pg:
container_name: pg-container
image: postgres:11.5
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
- pgstorage
ports:
- "5432:5432"
volumes:
pgstorage:
002-setup.sh
#!/bin/sh
# Remove last line "shared_preload_libraries='citus'"
sed -i '$ d' ${PGDATA}/postgresql.conf
cat <<EOT >> ${PGDATA}/postgresql.conf
shared_preload_libraries='pg_cron'
cron.database_name='${POSTGRES_DB:-postgres}'
EOT
# Required to load pg_cron
pg_ctl restart
003-main.sql
CREATE EXTENSION pg_cron;
From what I can see you are not installing pg_cron anywhere. Since it is not packaged with the default Postgres Docker image you will have to care of that.
For example by extending the Image and using a build entry in your docker-compose.yml.
# Dockerfile relative to docker-compose.yml
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
version: '3.7'
services:
pg:
container_name: pg-container
build: .
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
This worked for me - it probably needs some more optimization.
The proposed solution didn't work with a newly created container for me. So, I did it like this:
Docker file
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
RUN cd / && \
rm -rf /pg_cron && \
apt-get remove -y git build-essential postgresql-server-dev-11 && \
apt-get autoremove --purge -y && \
apt-get clean && \
apt-get purge
COPY init-db /docker-entrypoint-initdb.d
init-db/pg-cron.sh
#!/usr/bin/env bash
# use same db as the one from env
dbname="$POSTGRES_DB"
# create custom config
customconf=/var/lib/postgresql/data/custom-conf.conf
echo "" > $customconf
echo "shared_preload_libraries = 'pg_cron'" >> $customconf
echo "cron.database_name = '$dbname'" >> $customconf
chown postgres $customconf
chgrp postgres $customconf
# include custom config from main config
conf=/var/lib/postgresql/data/postgresql.conf
found=$(grep "include = '$customconf'" $conf)
if [ -z "$found" ]; then
echo "include = '$customconf'" >> $conf
fi
Also, you can place other init files into init-db directory.
Docker compose file
version: '3.7'
services:
postgres:
container_name: your-container
build: .
environment:
POSTGRES_DB: "your_db"
POSTGRES_USER: "your_user"
POSTGRES_PASSWORD: "your_user"
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
pgdata:
driver: local
For those who are looking for a ready image, please try the following:
docker pull ramazanpolat/postgres_cron:11

Looking for a docker image to containerized my spark scala streaming jobs

I am new to Spark(3.0.0_preview) and Scala(SBT). I have written a spark streaming job that I can run successfully on my local from my IDE
Now, I a looking for a way to dockerize the code so that I can run it with my docker-compose that builds the spark cluster
My docker-compose:
version: "3.3"
services:
spark-master:
image: rd/spark:latest
container_name: spark-master
hostname: spark-master
ports:
- "8080:8080"
- "7077:7077"
networks:
- spark-network
environment:
- "SPARK_LOCAL_IP=spark-master"
- "SPARK_MASTER_PORT=7077"
- "SPARK_MASTER_WEBUI_PORT=8080"
command: "/start-master.sh"
spark-worker:
image: rd/spark:latest
depends_on:
- spark-master
ports:
- 8080
networks:
- spark-network
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "SPARK_WORKER_WEBUI_PORT=8080"
command: "/start-worker.sh"
networks:
spark-network:
driver: bridge
ipam:
driver: default
Docker Files:
FROM openjdk:8-alpine
RUN apk --update add wget tar bash
RUN wget http://apache.mirror.anlx.net/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz
RUN tar -xzf spark-3.0.0-preview2-bin-hadoop2.7.tgz && \
mv spark-3.0.0-preview2-bin-hadoop2.7 /spark && \
rm spark-3.0.0-preview2-bin-hadoop2.7.tgz
COPY start-master.sh /start-master.sh
COPY start-worker.sh /start-worker.sh
This seems a simple request but I am having a hard time finding good documentation on it.
Got it working with the following:
project-structure:
project
src
build.sbt
Dockerfile
Dockerfile-app
docker-compose.yml
docker-compose.yml
version: "3.3"
services:
spark-scala-env:
image: app/spark-scala-env:latest
build:
context: .
dockerfile: Dockerfile
app-spark-scala:
image: app/app-spark-scala:latest
build:
context: .
dockerfile: Dockerfile-app
spark-master:
image: app/app-spark-scala:latest
container_name: spark-master
hostname: localhost
ports:
- "8080:8080"
- "7077:7077"
networks:
- spark-network
environment:
- "SPARK_LOCAL_IP=spark-master"
- "SPARK_MASTER_PORT=7077"
- "SPARK_MASTER_WEBUI_PORT=8080"
command: ["sh", "-c", "/spark/bin/spark-class org.apache.spark.deploy.master.Master --ip $${SPARK_LOCAL_IP} --port $${SPARK_MASTER_PORT} --webui-port $${SPARK_MASTER_WEBUI_PORT}"]
spark-worker:
image: app/app-spark-scala:latest
hostname: localhost
depends_on:
- spark-master
ports:
- 8080
networks:
- spark-network
#network_mode: host
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "SPARK_WORKER_WEBUI_PORT=8080"
- "SPARK_WORKER_CORES=2"
command: ["sh", "-c", "/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port $${SPARK_WORKER_WEBUI_PORT} $${SPARK_MASTER}"]
app-submit-job:
image: app/app-spark-scala:latest
ports:
- "4040:4040"
environment:
- "SPARK_APPLICATION_MAIN_CLASS=com.app.spark.TestAssembly"
- "SPARK_MASTER=spark://spark-master:7077"
- "APP_PACKAGES=org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0-preview2"
- "APP_JAR_LOC=/app/target/scala-2.12/app_spark_scala-assembly-0.2.jar"
hostname: localhost
networks:
- spark-network
volumes:
- ./appdata:/appdata
command: ["sh", "-c", "/spark/bin/spark-submit --packages $${APP_PACKAGES} --class $${SPARK_APPLICATION_MAIN_CLASS} --master $${SPARK_MASTER} $${APP_JAR_LOC}"]
networks:
spark-network:
driver: bridge
ipam:
driver: default
Dockerfile
FROM openjdk:8-alpine
ARG SPARK_VERSION
ARG HADOOP_VERSION
ARG SCALA_VERSION
ARG SBT_VERSION
ENV SPARK_VERSION=${SPARK_VERSION:-3.0.0-preview2}
ENV HADOOP_VERSION=${HADOOP_VERSION:-2.7}
ENV SCALA_VERSION ${SCALA_VERSION:-2.12.8}
ENV SBT_VERSION ${SBT_VERSION:-1.3.4}
RUN apk --update add wget tar bash
RUN \
echo "$SPARK_VERSION $HADOOP_VERSION" && \
echo http://apache.mirror.anlx.net/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz && \
wget http://apache.mirror.anlx.net/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz
RUN tar -xzf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz && \
mv spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} /spark && \
rm spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz
RUN \
echo "$SCALA_VERSION $SBT_VERSION" && \
mkdir -p /usr/lib/jvm/java-1.8-openjdk/jre && \
touch /usr/lib/jvm/java-1.8-openjdk/jre/release && \
apk add -U --no-cache bash curl && \
curl -fsL http://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz | tar xfz - -C /usr/local && \
ln -s /usr/local/scala-$SCALA_VERSION/bin/* /usr/local/bin/ && \
scala -version && \
scalac -version
RUN \
curl -fsL https://github.com/sbt/sbt/releases/download/v$SBT_VERSION/sbt-$SBT_VERSION.tgz | tar xfz - -C /usr/local && \
$(mv /usr/local/sbt-launcher-packaging-$SBT_VERSION /usr/local/sbt || true) \
ln -s /usr/local/sbt/bin/* /usr/local/bin/ && \
sbt sbt-version || sbt sbtVersion || true
Dockerfile-app
FROM app/spark-scala-env:latest
WORKDIR /app
# Pre-install base libraries
ADD build.sbt /app/
ADD project/plugins.sbt /app/project/
ADD src/. /app/src/
RUN sbt update
RUN sbt clean assembly
Commands:
Build the docker image with spark and scala env. i.e docker-compose build spark-scala-env
Build the image with the app jar. i.e docker-compose build app-spark-scala
Bring the spark env container i.e docker-compose up -d --scale spark-worker=2 spark-worker
Submit the job via docker-compose up -d app-submit-job

Cannot connect Lumen with PostgreSQL on Docker

I have Lumen with PostgreSQL dockerized with docker-compose. Both works like a charm but I cannot make a migration or insert data because Lumen container is not connected to the PostgreSQL one, it appears the following error:
SQLSTATE[08006] [7] could not connect to server: Connection refused.Is
the server running on host "postgresql" (172.18.0.3) and accepting
TCP/IP connections on port 9906?
Dockerfile
FROM php:7-apache
RUN apt-get upgrade -y \
&& apt-get update -y \
&& apt-get install git-core libzip-dev openssl unzip zip libpq-dev postgresql-client -y \
&& apt-get clean \
&& docker-php-ext-install mbstring pdo pdo_pgsql pgsql
# Apache Config
ENV APACHE_DOCUMENT_ROOT=/var/www/html/public
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
RUN a2enmod rewrite
# Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
WORKDIR /var/www/html
EXPOSE 8100
docker-compose.yml
version: '3.5'
services:
app:
build: .
container_name: form_app
depends_on:
- postgresql
ports:
- "8100:80"
volumes:
- .:/var/www/html/
stdin_open: true
tty: true
postgresql:
image: postgres
restart: always
container_name: form_pgsql
environment:
POSTGRES_USER: formularios
POSTGRES_PASSWORD: formularios
POSTGRES_DB: formularios
ports:
- "9906:5432"
volumes:
- pgdata:/var/lib/postgresql/data
admin:
image: adminer
container_name: form_adminer
ports:
- "8880:8080"
volumes:
pgdata:
.env
APP_NAME=Lumen
APP_ENV=local
APP_KEY=feZh5izBbOLIek27RQip9ciwTdRHPV7R
APP_DEBUG=true
APP_URL=http://localhost:8100
APP_TIMEZONE=UTC
LOG_CHANNEL=stack
LOG_SLACK_WEBHOOK_URL=
DB_CONNECTION=pgsql
DB_HOST=postgresql
DB_PORT=9906
DB_DATABASE=formularios
DB_USERNAME=formularios
DB_PASSWORD=formularios
CACHE_DRIVER=file
QUEUE_CONNECTION=sync