pg_dump server and pg_dump version mismatch in docker - postgresql

When I run the command psql --version within the railsApp container, I get 9.4.12 and when I run the same within the postgres container, I get 9.6.2. How can I get the versions to match?
I am getting the following error when I try to do a migration on Rails App which does a pg_dump sql import.
pg_dump: server version: 9.6.2; pg_dump version: 9.4.12
pg_dump: aborting because of server version mismatch
rails aborted!
Here's my Docker-compose.yml file:
version: "2.1"
services:
railsApp:
build:
context: ./
ports:
- "3000:3000"
links:
- postgres
volumes:
- .:/app
postgres:
image: postgres:9.6
ports:
- "5432:5432"
volumes:
- ./.postgres:/var/lib/postgresql
The Dockerfile:
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
FROM ruby:2.3.3
# setup /app as our working directory
RUN mkdir /app
WORKDIR /app
# Replace shell with bash so we can source files
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
# Set debconf to run non-interactively
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Install base dependencies
RUN apt-get update && apt-get install -y -q --no-install-recommends \
apt-transport-https \
build-essential \
ca-certificates \
curl \
git \
libssl-dev \
python \
rsync \
software-properties-common \
wget \
postgresql-client \
graphicsmagick \
&& rm -rf /var/lib/apt/lists/*
# Install node and npm with nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
ENV NVM_DIR=/root/.nvm
ENV NODE_VERSION v7.2.1
ENV NODE_PATH $NVM_DIR/versions/node/$NODE_VERSION
ENV PATH $NODE_PATH/bin:./node_modules/.bin:$PATH
RUN source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
# Install our ruby dependencies
ADD Gemfile Gemfile.lock /app/
RUN bundle install
# copy the rest of our code over
ADD . /app
ENV RAILS_ENV development
ENV SECRET_KEY_BASE a6bdc5f788624f00b68ff82456d94bf81bb50c2e114b2be19af2e6a9b76f9307b11d05af4093395b0471c4141b3cd638356f888e90080f8ae60710f992beba8f
# Expose port 3000 to the Docker host, so we can access it from the outside.
EXPOSE 3000
# Set the default command to run our server on port 3000
CMD ["rails", "server", "-p", "3000", "-b", "0.0.0.0"]

The same issue for me, I used an alternative way to take a dump,
First I access the terminal and run the pd_dump inside docker and copied the file from docker to host.
Below are the commands
docker exec -it <container-id> /bin/bash # accessing docker terminal
pg_dump > ~/dump # taking dump inside the docker
docker cp <container-id>:/root/dump ~/dump #copying the dump files to host
Hope the above solution helps.

The easiest approach is to use the correct postgres version in the docker-compose. Change:
postgres:
image: postgres:9.6
To:
postgres:
image: postgres:9.4.2
All available versions here.

Related

running elixir on buildkite with docker-compose fails with dependencies

i have the following dockerfile for an elixir+phoenix app
FROM elixir:latest as build_base
RUN apt-get -y update
RUN apt-get -y install inotify-tools curl
ARG TARGETARCH
RUN if [ ${TARGETARCH} = arm64 ]; then \
curl -L -o /tmp/dart-sass.tar.gz https://github.com/sass/dart-sass/releases/download/1.54.5/dart-sass-1.54.5-linux-${TARGETARCH}.tar.gz \
;else \
curl -L -o /tmp/dart-sass.tar.gz https://github.com/sass/dart-sass/releases/download/1.54.5/dart-sass-1.54.5-linux-x64.tar.gz \
;fi
RUN tar -xvf /tmp/dart-sass.tar.gz -C /tmp
RUN mv /tmp/dart-sass/sass /usr/local/bin/sass
RUN mkdir -p /app
WORKDIR /app
COPY mix.* ./
RUN mix local.hex --force
RUN mix archive.install hex phx_new --force
RUN mix local.rebar --force
RUN mix deps.clean --all
RUN mix deps.get
RUN mix --version
RUN mix deps.compile
COPY assets assets
COPY vendor vendor
COPY lib lib
COPY config config
COPY priv priv
COPY test test
RUN mix compile
the docker-compose file looks like the following
services:
web:
build:
context: .
dockerfile: Dockerfile
target: build_base
volumes:
- ./:/app
ports:
- "80:80"
command: mix phx.server
I'm trying to run docker-compose as part of the build step in buildkite, this is an extract of the step in buildkite
- label: "run web"
key: "web"
commands:
- mix phx.server
plugins:
- docker-compose#v4.9.0:
run: web
config: docker-compose.yml
however when running web i see everything happens properly including the package installation, however when running the application i see the following error
web_1 | Unchecked dependencies for environment dev:
web_1 | * telemetry_metrics (Hex package)
web_1 | the dependency is not available, run "mix deps.get"
and the list goes on and on, this works fine on my local machine, its only when running on buildkite. does anyone have any idea on how to fix this ?

docker-compose environment variables inside container

I have a small python app developed with docker containers.
My setup is:
Dockerfile
FROM python:3
ARG http_proxy
ARG https_proxy
ENV http_proxy ${http_proxy}
ENV https_proxy ${https_proxy}
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN apt-get update
RUN apt install -y vim screen
RUN \
echo 'alias py="/opt/venv/bin/python"' >> /root/.bashrc && \
echo 'alias ls="ls --color=auto"' >> /root/.bashrc && \
echo 'PS1="\u#\h:\[\e[33m\]\w\[\e[0m\]\$ "' >> /root/.bashrc
RUN python3 -m venv $VIRTUAL_ENV
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
docker-compose.yml
version: '3.8'
x-dev:
&proxy_conf
http_proxy: "${HTTP_PROXY}"
https_proxy: "${HTTPS_PROXY}"
services:
py_service:
container_name: ${APP_NAME}
build:
context: .
dockerfile: Dockerfile
args: *proxy_conf
image: ${APP_NAME}_img
volumes:
- '.:/app'
restart: always
command: tail -f /dev/null
.env
HTTP_PROXY=<http_proxy_server_here>
HTTPS_PROXY=<https_proxy_server_here>
APP_NAME=python_app
The problem is if the proxy server has changed i need to rebuild the image and i don't want that(as a last result maybe i will do it).
What i'm trying to do is change the proxy environment variables inside the container but i don't find the file where the env is stored.
The container OS version is:
[root#5b1b77079e10 ~ >>>] $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
You should only need to recreate containers, not rebuild the image. I assume you are doing something like this to get everything up initially:
docker-compose build
docker-compose up -d
Then I assume you are updating you .env file, once you do that you should be able to just do the following for your container to pick up the change:
docker-compose down
docker-compose up -d
You should not need to do a docker-compose build again.

Two docker images into a single docker image [duplicate]

This question already has answers here:
Docker: Combine multiple images
(4 answers)
Closed 2 years ago.
I have a requirement to create a single docker image by merging both "RabbitMQ-management" and "MongoDB" images. Is it possible to do it?
I've tried with below Dockerfile and can able to access RabbitMQ, but mongodb is not working. Could you please help me.
# cat Dockerfile
FROM rabbitmq:3.7.17
RUN rabbitmq-plugins enable --offline rabbitmq_management
# extract "rabbitmqadmin" from inside the "rabbitmq_management-X.Y.Z.ez" plugin zipfile
# see https://github.com/docker-library/rabbitmq/issues/207
RUN apt-get update && apt-get install -y gnupg2
RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 0C49F3730359A14518585931BC711F9BA15703C6 && \
gpg --export $GPG_KEYS > /etc/apt/trusted.gpg.d/mongodb.gpg
ARG MONGO_PACKAGE=mongodb-org
ARG MONGO_REPO=repo.mongodb.org
ENV MONGO_PACKAGE=${MONGO_PACKAGE} MONGO_REPO=${MONGO_REPO}
ENV MONGO_MAJOR 3.4
ENV MONGO_VERSION 3.4.18
RUN echo "deb http://$MONGO_REPO/apt/debian jessie/${MONGO_PACKAGE%-unstable}/$MONGO_MAJOR main" | tee "/etc/apt/sources.list.d/${MONGO_PACKAGE%-unstable}.list"
RUN echo "/etc/apt/sources.list.d/${MONGO_PACKAGE%-unstable}.list"
RUN apt-get update
RUN apt-get install -y ${MONGO_PACKAGE}=$MONGO_VERSION
RUN set -eux; \
erl -noinput -eval ' \
{ ok, AdminBin } = zip:foldl(fun(FileInArchive, GetInfo, GetBin, Acc) -> \
case Acc of \
"" -> \
case lists:suffix("/rabbitmqadmin", FileInArchive) of \
true -> GetBin(); \
false -> Acc \
end; \
_ -> Acc \
end \
end, "", init:get_plain_arguments()), \
io:format("~s", [ AdminBin ]), \
init:stop(). \
' -- /plugins/rabbitmq_management-*.ez > /usr/local/bin/rabbitmqadmin; \
[ -s /usr/local/bin/rabbitmqadmin ]; \
chmod +x /usr/local/bin/rabbitmqadmin; \
apt-get update; apt-get install -y --no-install-recommends python ca-certificates; rm -rf /var/lib/apt/lists/*; \
rabbitmqadmin --version
EXPOSE 15671 15672 27017
I'm using below command to run RabbitMQ (it is working)
# docker run -it -p 15672:15672 -p 5672:5672 --hostname my-rabbitmq image
Command to start Mongodb container (it is not working)
docker run -d -v /tmp/mongodb:/data/db -p 27017:27017 image mongod
It is not impossible to achieve what you are asking, but I would say it's not the correct way of doing it. Docker provides isolation, and states that it should be one process per container. (should, not must).
If you follow this concept, it's easy to use other tools to bring up multiple containers together. For example, instead of complicated Dockerfile that merge two images you can specify a nice docker-compose.yml:
version: '3'
services:
mongo:
image: mongod
volumes:
- /tmp/mongodb:/data/db
ports:
- "27017:27017"
rabbitmq:
image: rabbitmq
ports:
- "15672:15672"
- "5672:5672"
depends_on:
- mongo
Which is more readable IMO, and probably achieve the same outcome.

No existing local cluster is suitable error using pg_dump in docker

In my docker project with postgres 9.6 I need to add support for uploading of big sql dumps.
I added pg_dump by adding in my web/Dockerfile.yml file line:
postgresql-client-common \
as it contains pg_dump.
After building the app I enter bash and
# uname -a
Linux ff5146f21dbd 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux
root#ff5146f21dbd:/var/www/lprods_docker_root# pg_dump
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
root#ff5146f21dbd:/var/www/lprods_docker_root# locate -i pg_dump
Last command outputs nothing
web/Dockerfile.yml contains :
FROM php:7.1-apache
RUN apt-get update && \
apt-get install -y \
python \
libfreetype6-dev \
libwebp-dev \
libjpeg62-turbo-dev \
libpng-dev \
libzip-dev \
nano \
git-core \
curl \
build-essential \
openssl \
libssl-dev \
libgmp-dev \
libldap2-dev \
libpq-dev \
netcat \
postgresql-client-common \
locate \
&& git clone https://github.com/nodejs/node.git \
&& cd node \
&& git checkout v12.0.0 \
&& ./configure \
&& make \
&& make install
RUN npm install cross-env
RUN docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-webp-dir=/usr/include/ --with-jpeg-dir=/usr/include/
RUN docker-php-ext-install gd pgsql pdo_pgsql zip gmp bcmath pcntl ldap sysvmsg exif \
&& a2enmod rewrite
COPY virtualhost.conf /etc/apache2/sites-enabled/000-default.conf
and docker-compose.yml:
version: '3'
services:
web:
build:
context: ./web # directory of web/Dockerfile.yml
dockerfile: Dockerfile.yml
environment:
- APACHE_RUN_USER=#1000
# - APACHE_RUN_USER=www-data
container_name: lprods_web
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
ports:
- "8086:80"
working_dir: ${APP_PTH_CONTAINER}
db:
image: postgres:9.6.10-alpine
container_name: lprods_db
ports:
- '5433:5432'
restart: always
environment:
POSTGRES_USER: 'postgres'
POSTGRES_PASSWORD: '1'
POSTGRES_DB: 'wprods'
volumes:
- ./init:/docker-entrypoint-initdb.d/
phppgadmin:
image: dockage/phppgadmin:latest
environment:
- PHP_PG_ADMIN_SERVER_HOST=db
- PHP_PG_ADMIN_SERVER_PORT=5432
- PHP_PG_ADMIN_SERVER_DEFAULT_DB=postgres
container_name: lprods_phppgadmin
restart: always
ports:
- "8087:80"
- "443:443"
links:
- db
composer:
image: composer:1.6
container_name: lprods_composer
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
working_dir: ${APP_PTH_CONTAINER}
command: composer install --ignore-platform-reqs
Why error and how to fix it?
MODIFIED :
I have
# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
But checking version I got error :
# /usr/bin/pg_dump -v
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
The same error I got when running psql command(actually I need it):
/usr/bin/psql -h localhost -d wprods -U postgres -W -f "/var/www/lprods_docker_root/wprods_2017_10_23.sql"
Error: You must install at least one postgresql-client-<version> package
What is wrong with my configuration ?
MODIFIED 2:
In my docker-compose.yml I have :
db:
image: postgres:9.6.10-alpine
...
volumes:
- ./init:/docker-entrypoint-initdb.d/
Is it?
command :
locate -i postgresql.conf
outputs nothing
find /usr -name "postgresql.conf"
Also outputs nothing.
Also I have :
# ps -ef | grep postgres
root 28 21 0 08:40 pts/0 00:00:00 grep postgres
What did I miss ?
postgresql-client-common is a Ubuntu-specific package of scripts that allows users to work with multiple versions of Postgres (or multiple database clusters). It provies several scripts and symlinks that essentially override the functions of "real" postgres utilities (like pg_dump):
root#foo:/usr/bin# ls -al | grep pg_dump
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dump -> ../share/postgresql-common/pg_wrapper
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dumpall -> ../share/postgresql-common/pg_wrapper
You may need to also install postgresql-client-10 to make pg_dump work.
Note also that you have only installed the client tools. If you need to get a fully-functional PostgreSQL database up and running, I believe you will also need to install the postgresql-10 package

Where is the Postgres username/password being created in this Dockerfile?

So I was following this tutorial:
https://realpython.com/blog/python/django-development-with-docker-compose-and-machine/
I have everything up and running, however theres a few things going on that I'm not able to follow or understand.
In the main Docker-Compose we have:
web:
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
- redis:redis
volumes:
- /usr/src/app
- /usr/src/app/static
env_file: .env
command: /usr/local/bin/gunicorn docker_django.wsgi:application -w 2 -b :8000
postgres:
restart: always
image: postgres:latest
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data/
You will notice there is a env_file containing:
DB_NAME=postgres
DB_USER=postgres
DB_PASS=postgres
My question is when is the postgres user and password being set? If I run this docker-compose everything works, meaning the web app can pass credentials to the postgress database and establish a connection. I'm not able to follow however, where those credentials are being set in the first place.
I was assuming in the base postgres Dockerfile, there would be some instruction to set the database name, username and password. I do not it though. Here is a copy of the base postgres Dockerfile below.
# vim:set ft=dockerfile:
FROM debian:jessie
# explicitly set user/group IDs
RUN groupadd -r postgres --gid=999 && useradd -r -g postgres --uid=999 postgres
# grab gosu for easy step-down from root
ENV GOSU_VERSION 1.7
RUN set -x \
&& apt-get update && apt-get install -y --no-install-recommends ca-certificates wget && rm -rf /var/lib/apt/lists/* \
&& wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
&& wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture).asc" \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
&& gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
&& rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& gosu nobody true \
&& apt-get purge -y --auto-remove ca-certificates wget
# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
RUN mkdir /docker-entrypoint-initdb.d
RUN apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
ENV PG_MAJOR 9.6
ENV PG_VERSION 9.6.1-1.pgdg80+1
RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list
RUN apt-get update \
&& apt-get install -y postgresql-common \
&& sed -ri 's/#(create_main_cluster) .*$/\1 = false/' /etc/postgresql-common/createcluster.conf \
&& apt-get install -y \
postgresql-$PG_MAJOR=$PG_VERSION \
postgresql-contrib-$PG_MAJOR=$PG_VERSION \
&& rm -rf /var/lib/apt/lists/*
# make the sample config easier to munge (and "correct by default")
RUN mv -v /usr/share/postgresql/$PG_MAJOR/postgresql.conf.sample /usr/share/postgresql/ \
&& ln -sv ../postgresql.conf.sample /usr/share/postgresql/$PG_MAJOR/ \
&& sed -ri "s!^#?(listen_addresses)\s*=\s*\S+.*!\1 = '*'!" /usr/share/postgresql/postgresql.conf.sample
RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
ENV PATH /usr/lib/postgresql/$PG_MAJOR/bin:$PATH
ENV PGDATA /var/lib/postgresql/data
VOLUME /var/lib/postgresql/data
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 5432
CMD ["postgres"]
Not sure which postgres image are you using.
If you look at the official postgres image for complete information. It allows user to specify the environment variables for the below ones and these can be easily overridden at run time.
POSTGRES_PASSWORD
POSTGRES_USER
PGDATA
POSTGRES_DB
POSTGRES_INITDB_ARGS
Environment variables can be overridden using below three methods depending on your case.
Run Image: If running docker image directly, use below to include environment variable in docker run with -e K=V. Please refer documentation for more details here
docker run -e POSTGRES_PASSWORD=secrect -e POSTGRES_USER=postgres <other options> image/name
Dockerfile: If you need to specify the environment variable in Dockerfile, specify as mentioned below. Please refer documentation for more details here
ENV POSTGRES_PASSWORD=secrect
ENV POSTGRES_USER=postgres
docker-compose: If you need to specify the environment variable in docker-compose.yml, specify as below. Please refer documentation for more details here
web:
environment:
- POSTGRES_PASSWORD=secrect
- POSTGRES_USER=postgres
Hope this is useful.
I think the postgres user and password is being set on entrypoint, like in line 23 on official image entrypoint.
https://github.com/docker-library/postgres/blob/e4942cb0f79b61024963dc0ac196375b26fa60dd/9.6/docker-entrypoint.sh
Can you check your entrypoint?
All you need is
docker run -e POSTGRES_PASSWORD=123456 postgres
In the same way you can set up the username like below:
docker run -e POSTGRES_USERNAME=xyz postgres
Are you using the -v mounted folder? If so, you need to remove the entire pgdata/ folder you created previously so that the postgres can re-create itself using the new environment variables.