I am super new to Docker and my first task as an intern is to fix some problems related to docker in a Laravel + Postgres + pgadmin + redis containerized project. I'm running everything on WSL2 Ubuntu 20.04 (since I only have Windows 10 Home)
My questions are simple I believe.
Why postgres and pgadmin volumes names are set with a '.' in the beggining?? And can someone spot a problem with the pgadmin service? It's not working after docker-compose up -d, do I have to do anything else to set it up?
postgres-app:
image: postgres:12
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: recadastramento
volumes:
- .\postgres:/var/lib/postgresql/data
networks:
- app-network
pgadmim-app:
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: "noriakiakiba#gmail.com"
PGADMIN_DEFAULT_PASSWORD: "123456"
ports:
- "16543:80"
depends_on:
- postgres-app
volumes:
- .\pgadmin:/var/lib/pgadmin
networks:
- app-network
networks:
app-network:
driver: bridge
My Dockerfile
# Dockerfile
FROM php:7.4.1-apache
#se nao criar essa pasta dá erro ao instalar openjdk
RUN mkdir /usr/share/man/man1/
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
git-core \
openssl \
libssl-dev \
autoconf \
build-essential \
apt-utils \
zlib1g-dev \
libzip-dev \
unzip \
zip \
libmagick++-dev \
libgraphicsmagick1-dev \
libmagickwand-dev \
libpq-dev \
libfreetype6-dev \
libjpeg62-turbo-dev \
libpng-dev \
libwebp-dev \
libxpm-dev \
nano \
gcc \
g++ \
curl \
imagemagick \
git \
zip \
unzip \
libcurl4-openssl-dev \
make \
ant \
apt-transport-https \
ca-certificates \
dirmngr \
wget \
locales
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
locale-gen
# 2. apache configs + document root
ENV APACHE_DOCUMENT_ROOT=/var/www/html/public
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
# 3. mod_rewrite for URL rewrite and mod_headers for .htaccess extra headers like Access-Control-Allow-Origin-
RUN a2enmod rewrite headers
# 4. start with base php config, then add extensions
RUN mv "$PHP_INI_DIR/php.ini-development" "$PHP_INI_DIR/php.ini"
# Configure php extensions
RUN docker-php-ext-configure gd --with-freetype --with-jpeg
# Install php extensions
RUN docker-php-ext-install \
bcmath \
calendar \
curl \
exif \
gd \
iconv \
intl \
pdo \
pdo_pgsql \
pcntl \
tokenizer \
xml \
zip \
json \
soap
# Install Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
#RUN git clone https://github.com/nodejs/node.git \
# && cd node \
# && ./configure \
# && make \
# && sudo make install
RUN chown -R www-data:www-data /var/www/html
RUN echo 'memory_limit = -1' >> /usr/local/etc/php/conf.d/docker-php-memlimit.ini;
# Clear package lists
RUN apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/doc/*
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash - &&\
apt-get -y install nodejs
RUN pecl install xdebug \
&& docker-php-ext-enable xdebug \
&& echo "xdebug.mode=debug" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.client_host = host.docker.internal" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
RUN pecl install imagick \
&& echo "extension=imagick.so" >> /usr/local/etc/php/php.ini
Ok, as recommended I executed sudo docker-compose up and got the following errors (the last part of postgres is repetead forever):
postgres-app_1 | chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
laravel-app_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.3. Set the 'ServerName' directive globally to suppress this message
postgres-app_1 | The files belonging to this database system will be owned by user "postgres".
postgres-app_1 | This user must also own the server process.
postgres-app_1 |
postgres-app_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres-app_1 | The default database encoding has accordingly been set to "UTF8".
postgres-app_1 | The default text search configuration will be set to "english".
postgres-app_1 |
postgres-app_1 | Data page checksums are disabled.
postgres-app_1 |
postgres-app_1 | fixing permissions on existing directory /var/lib/postgresql/data ... initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
pgadmim-app_1 | Traceback (most recent call last):
pgadmim-app_1 | File "/pgadmin4/run_pgadmin.py", line 4, in <module>
pgadmim-app_1 | from pgAdmin4 import app
pgadmim-app_1 | File "/pgadmin4/pgAdmin4.py", line 94, in <module>
pgadmim-app_1 | WARNING: Failed to set ACL on the directory containing the configuration database:
pgadmim-app_1 | [Errno 1] Operation not permitted: '/var/lib/pgadmin'
pgadmim-app_1 | HINT : You may need to manually set the permissions on
pgadmim-app_1 | /var/lib/pgadmin to allow pgadmin to write to it.
pgadmim-app_1 | app = create_app()
pgadmim-app_1 | File "/pgadmin4/pgadmin/__init__.py", line 256, in create_app
pgadmim-app_1 | create_app_data_directory(config)
pgadmim-app_1 | File "/pgadmin4/pgadmin/setup/data_directory.py", line 90, in create_app_data_directory
pgadmim-app_1 | os.chmod(config.SESSION_DB_PATH, 0o700)
pgadmim-app_1 | PermissionError: [Errno 1] Operation not permitted: '/var/lib/pgadmin/sessions'
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [1] [INFO] Starting gunicorn 19.9.0
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [1] [INFO] Listening at: http://[::]:80 (1)
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [1] [INFO] Using worker: threads
pgadmim-app_1 | /usr/local/lib/python3.9/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
pgadmim-app_1 | return io.open(fd, *args, **kwargs)
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [87] [INFO] Booting worker with pid: 87
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [87] [ERROR] Exception in worker process
pgadmim-app_1 | Traceback (most recent call last):
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
pgadmim-app_1 | worker.init_process()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/gthread.py", line 104, in init_process
pgadmim-app_1 | super(ThreadWorker, self).init_process()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 129, in init_process
pgadmim-app_1 | self.load_wsgi()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
pgadmim-app_1 | self.wsgi = self.app.wsgi()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
pgadmim-app_1 | self.callable = self.load()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
pgadmim-app_1 | return self.load_wsgiapp()
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
pgadmim-app_1 | return util.import_app(self.app_uri)
pgadmim-app_1 | File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 350, in import_app
pgadmim-app_1 | __import__(module)
pgadmim-app_1 | File "/pgadmin4/run_pgadmin.py", line 4, in <module>
pgadmim-app_1 | from pgAdmin4 import app
pgadmim-app_1 | File "/pgadmin4/pgAdmin4.py", line 94, in <module>
pgadmim-app_1 | app = create_app()
pgadmim-app_1 | File "/pgadmin4/pgadmin/__init__.py", line 256, in create_app
pgadmim-app_1 | create_app_data_directory(config)
pgadmim-app_1 | File "/pgadmin4/pgadmin/setup/data_directory.py", line 90, in create_app_data_directory
pgadmim-app_1 | os.chmod(config.SESSION_DB_PATH, 0o700)
pgadmim-app_1 | PermissionError: [Errno 1] Operation not permitted: '/var/lib/pgadmin/sessions'
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [87] [INFO] Worker exiting (pid: 87)
pgadmim-app_1 | WARNING: Failed to set ACL on the directory containing the configuration database:
pgadmim-app_1 | [Errno 1] Operation not permitted: '/var/lib/pgadmin'
pgadmim-app_1 | HINT : You may need to manually set the permissions on
pgadmim-app_1 | /var/lib/pgadmin to allow pgadmin to write to it.
docker_postgres-app_1 exited with code 1
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [1] [INFO] Shutting down: Master
pgadmim-app_1 | [2021-01-28 22:57:21 +0000] [1] [INFO] Reason: Worker failed to boot.
docker_pgadmim-app_1 exited with code 3
postgres-app_1 | chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
postgres-app_1 | The files belonging to this database system will be owned by user "postgres".
postgres-app_1 | This user must also own the server process.
postgres-app_1 |
postgres-app_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres-app_1 | The default database encoding has accordingly been set to "UTF8".
postgres-app_1 | The default text search configuration will be set to "english".
postgres-app_1 |
postgres-app_1 | Data page checksums are disabled.
postgres-app_1 |
postgres-app_1 | initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
postgres-app_1 | fixing permissions on existing directory /var/lib/postgresql/data ... chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
I searched for this error and apparently I need to run chown -R 5050:5050 /var/lib in the host in order for pgadmin to run on localhost:16543, but what I want is to fix this automatically (maybe with the Dockerfile?)
I have a similar docker-compose.yml file, but I use named volumes for my db service. Only for my pgadmin service I use bind mount ./pgadmin:/var/lib/pgadmin, to bind the directory ./pgadmin in my windows host machine to the directory /var/lib/pgadmin in container. I find it useful for pgadmin service, because when we generate a backupfile of a database/server, we can take it directly from ./pgadmin in host machine.
Following is a compose file that works perfectly fine for me.
docker-compose.yml
version: "3.7"
services:
db:
image: postgres:12
container_name: db
volumes:
- db_data:/var/lib/postgresql/data
ports:
- "5432:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=dbname
pgadmin:
image: dpage/pgadmin4
container_name: pgadmin
depends_on:
- db
volumes:
- ./pgadmin:/var/lib/pgadmin
ports:
- "5555:80"
environment:
PGADMIN_DEFAULT_EMAIL: pgadmin#domain.de
PGADMIN_DEFAULT_PASSWORD: pgadmin
volumes:
db_data:
permission of directory /var/lib/pgadmin in pgadmin container:
/var/lib $ ls -l
total 20
drwxr-xr-x 2 root root 4096 Dec 16 10:31 apk
drwxr-xr-x 2 root root 4096 Dec 16 10:31 misc
drwxrwxrwx 1 pgadmin pgadmin 0 Jan 29 03:09 pgadmin
drwx------ 1 postfix root 4096 Jan 29 03:09 postfix
drwx--x--x 3 root root 4096 Jan 25 14:35 sudo
drwxr-xr-x 2 root root 4096 Dec 16 10:31 udhcpd
Related
I want to unit-test my app which uses a postgres database inside a docker.
EDIT: based on the suggested answer I modified the Dockerfile:
FROM postgres
USER postgres
RUN sleep 2 # remark 1
RUN initdb # remark 2
RUN sleep 3 # remark 1
RUN psql --host=localhost -l
The remarks are:
from this reference
Try putting a sleep in there and see if it's still a problem
from the docs:
The default postgres user and database are created in the entrypoint with initdb.
Here is the Dockerfile from the original question:
FROM postgres
COPY input.json .
RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb
#
# [ 1 ] run application on input.json
# [ 2 ] check db content after run
#
When I use the above Dockerfile I seem to be missing something:
(The errors from the edited version are the same)
$ docker build --tag host --file Dockerfile .
[+] Building 0.3s (7/7) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 125B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/postgres:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 40B 0.0s
=> CACHED [1/3] FROM docker.io/library/postgres 0.0s
=> [2/3] COPY input.json . 0.0s
=> ERROR [3/3] RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb 0.2s
------
> [3/3] RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb:
#7 0.188 createdb: error: connection to server at "localhost" (127.0.0.1), port 7654 failed: Connection refused
#7 0.188 Is the server running on that host and accepting TCP/IP connections?
#7 0.188 connection to server at "localhost" (::1), port 7654 failed: Cannot assign requested address
#7 0.188 Is the server running on that host and accepting TCP/IP connections?
------
Postgres database starts only after you create a container based on postgres image. docker build process doesn't start entrypoint script. You might need a bash script or CI pipeline where you firstly start postgres container and then use it in your unit tests
$ docker run --name mypg -p 5432:5432 -e POSTGRES_PASSWORD=mypgpass -d postgres:9
# copy a script to the mypg container
$ docker cp run.sh mypg:/root/run.sh
# run the script
$ docker exec mypg bash /root/run.sh
...
# use postgres client on your host to connect to mypg container
$ PGPASSWORD="mypgpass" psql -U postgres -p 5432 -h localhost -c "select version()"
version
--------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 9.6.24 on x86_64-pc-linux-gnu (Debian 9.6.24-1.pgdg90+1), compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit
(1 row)
Postgres container docs
Postgres client authentication
EDIT:
By trying to run initdb, psql etc directly in Dockerfile, you are reinventing the docker-entrypoint.sh
During the build step of the postgres Docker you cannot run postgres commands. Postgres database will only be available after you run the Docker.
As specified in the postgres Docker documentation you can add customization to your postgres instance through scripts placed in docker-entrypoint-initdb.d directory.
Alternatively you could use a RUN directive to start the postgres database and after that run the postgres commands you want (making sure to wait for the DB to accept connections), as mentioned here.
Another side note, I personally avoid using real databases for unit testing applications. To me, it's always better to mock the database for unit tests, in python you can do this with unittest.mock.
Here is a complete answer based on the concepts of slava-kuravsky and mello:
$ docker build --tag host --file Dockerfile .
$ docker run -d -t --name pg host
$ docker exec pg bash run.sh
The script can do what I want, currently it only lists the databases:
$ cat run.sh # <--- copied during docker build
echo "Hello Postgres World"
psql --host=localhost -l
The Dockerfile does only initialization:
$ cat Dockerfile
FROM postgres
USER postgres
COPY run.sh . # <--- the testing script
RUN sleep 2 # <--- probably not needed anymore
RUN initdb
RUN sleep 3 # <--- probably not needed anymore
When I perform the three commands above I get what I expect 🙂 :
$ docker build --tag host --file Dockerfile .
[+] Building 7.2s (10/10) FINISHED
# ... omitted for brevity ...
$ docker run -d -t --name pg host
608ac7324e838924c9a5d0cfe65c8000e33350b86faf9df4511ef5fcf7440597
$ docker exec pg bash run.sh
Hello Postgres World
List of databases
Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges
-----------+----------+----------+------------+------------+------------+-----------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres +
| | | | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres +
| | | | | | | postgres=CTc/postgres
(3 rows)
In my docker project with postgres 9.6 I need to add support for uploading of big sql dumps.
I added pg_dump by adding in my web/Dockerfile.yml file line:
postgresql-client-common \
as it contains pg_dump.
After building the app I enter bash and
# uname -a
Linux ff5146f21dbd 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux
root#ff5146f21dbd:/var/www/lprods_docker_root# pg_dump
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
root#ff5146f21dbd:/var/www/lprods_docker_root# locate -i pg_dump
Last command outputs nothing
web/Dockerfile.yml contains :
FROM php:7.1-apache
RUN apt-get update && \
apt-get install -y \
python \
libfreetype6-dev \
libwebp-dev \
libjpeg62-turbo-dev \
libpng-dev \
libzip-dev \
nano \
git-core \
curl \
build-essential \
openssl \
libssl-dev \
libgmp-dev \
libldap2-dev \
libpq-dev \
netcat \
postgresql-client-common \
locate \
&& git clone https://github.com/nodejs/node.git \
&& cd node \
&& git checkout v12.0.0 \
&& ./configure \
&& make \
&& make install
RUN npm install cross-env
RUN docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-webp-dir=/usr/include/ --with-jpeg-dir=/usr/include/
RUN docker-php-ext-install gd pgsql pdo_pgsql zip gmp bcmath pcntl ldap sysvmsg exif \
&& a2enmod rewrite
COPY virtualhost.conf /etc/apache2/sites-enabled/000-default.conf
and docker-compose.yml:
version: '3'
services:
web:
build:
context: ./web # directory of web/Dockerfile.yml
dockerfile: Dockerfile.yml
environment:
- APACHE_RUN_USER=#1000
# - APACHE_RUN_USER=www-data
container_name: lprods_web
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
ports:
- "8086:80"
working_dir: ${APP_PTH_CONTAINER}
db:
image: postgres:9.6.10-alpine
container_name: lprods_db
ports:
- '5433:5432'
restart: always
environment:
POSTGRES_USER: 'postgres'
POSTGRES_PASSWORD: '1'
POSTGRES_DB: 'wprods'
volumes:
- ./init:/docker-entrypoint-initdb.d/
phppgadmin:
image: dockage/phppgadmin:latest
environment:
- PHP_PG_ADMIN_SERVER_HOST=db
- PHP_PG_ADMIN_SERVER_PORT=5432
- PHP_PG_ADMIN_SERVER_DEFAULT_DB=postgres
container_name: lprods_phppgadmin
restart: always
ports:
- "8087:80"
- "443:443"
links:
- db
composer:
image: composer:1.6
container_name: lprods_composer
volumes:
- ${APP_PATH_HOST}:${APP_PTH_CONTAINER}
working_dir: ${APP_PTH_CONTAINER}
command: composer install --ignore-platform-reqs
Why error and how to fix it?
MODIFIED :
I have
# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
But checking version I got error :
# /usr/bin/pg_dump -v
Warning: No existing local cluster is suitable as a default target. Please see man pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package
The same error I got when running psql command(actually I need it):
/usr/bin/psql -h localhost -d wprods -U postgres -W -f "/var/www/lprods_docker_root/wprods_2017_10_23.sql"
Error: You must install at least one postgresql-client-<version> package
What is wrong with my configuration ?
MODIFIED 2:
In my docker-compose.yml I have :
db:
image: postgres:9.6.10-alpine
...
volumes:
- ./init:/docker-entrypoint-initdb.d/
Is it?
command :
locate -i postgresql.conf
outputs nothing
find /usr -name "postgresql.conf"
Also outputs nothing.
Also I have :
# ps -ef | grep postgres
root 28 21 0 08:40 pts/0 00:00:00 grep postgres
What did I miss ?
postgresql-client-common is a Ubuntu-specific package of scripts that allows users to work with multiple versions of Postgres (or multiple database clusters). It provies several scripts and symlinks that essentially override the functions of "real" postgres utilities (like pg_dump):
root#foo:/usr/bin# ls -al | grep pg_dump
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dump -> ../share/postgresql-common/pg_wrapper
lrwxrwxrwx 1 root root 37 Feb 8 2018 pg_dumpall -> ../share/postgresql-common/pg_wrapper
You may need to also install postgresql-client-10 to make pg_dump work.
Note also that you have only installed the client tools. If you need to get a fully-functional PostgreSQL database up and running, I believe you will also need to install the postgresql-10 package
In Docker, I am trying to configure Postgres, get it up and running in another container, and link it to my users service, set up with the following structure:
docker-compose-dev.yml
services/
users/
manage.py
Dockerfile-dev
entrypoint.sh
project/
__init__.py
config.py
db/
create.sql
Dockerfile
docker-compose-dev.yml
version: '3.7'
services:
users:
build:
context: ./services/users
dockerfile: Dockerfile-dev
volumes:
- './services/users:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_APP=project/__init__.py
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#users-db:5432/users_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#users-db:5432/users_test
depends_on:
- users-db
users-db:
build:
context: ./services/users/project/db
dockerfile: Dockerfile
ports:
- 5435:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
Dockerfile-dev
# base image
FROM python:3.7.2-alpine
# install dependencies
RUN apk update && \
apk add --virtual build-deps gcc python-dev musl-dev && \
apk add postgresql-dev && \
apk add netcat-openbsd
# set working directory
WORKDIR /usr/src/app
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
Here I try to extend the official Postgres image by adding a SQL file to the "docker-entrypoint-initdb.d" directory in the container.
Dockerfile
# base image
FROM postgres:11.1-alpine
# run create.sql on init
ADD create.sql /docker-entrypoint-initdb.d
create.sql
CREATE DATABASE users_prod;
CREATE DATABASE users_dev;
CREATE DATABASE users_test;
Since the "users" service is dependent not only on the container being up and running but also the actual Postgres instance being up and healthy, I added an entrypoint.sh file to "users":
entrypoint.sh
#!/bin/sh
echo "Waiting for postgres..."
while ! nc -z users-db 5432; do
sleep 0.1
done
echo "PostgreSQL started"
python manage.py run -h 0.0.0.0
manage.py
from flask.cli import FlaskGroup
from project import app, db
cli = FlaskGroup(app)
#cli.command('recreate_db')
def recreate_db():
db.drop_all()
db.create_all()
db.session.commit()
if __name__ == '__main__':
cli()
config.py
import os
class BaseConfig:
"""Base configuration"""
TESTING = False
SQLALCHEMY_TRACK_MODIFICATIONS = False
class DevelopmentConfig(BaseConfig):
"""Development configuration"""
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL')
class TestingConfig(BaseConfig):
"""Testing configuration"""
TESTING = True
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_TEST_URL')
class ProductionConfig(BaseConfig):
"""Production configuration"""
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL')
I run and successfully build it with:
docker-compose -f docker-compose-dev.yml up -d --build
but I'm having some network issue. If I run:
docker-compose -f docker-compose-dev.yml logs
I get:
Attaching to dev2_users_1
users_1 | Waiting for postgres...
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
(...)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6ca5718c9867 dev2_users "/usr/src/app/entryp…" 2 minutes ago Up 2 minutes 0.0.0.0:5001->5000/tcp dev2_users_1
What am I missing in the code above?
You need to install bind-tools in Dockerfile-dev for Name does not resolve problem:
apk add bind-tools
Test:
/usr/src/app # nslookup users-db
nslookup: can't resolve '(null)': Name does not resolve
Name: users-db
Address 1: 172.20.0.2 stc_users-db_1_f435854a8c88.stc_default
/usr/src/app # apk add bind-tools
(1/4) Installing json-c (0.13.1-r0)
(2/4) Installing libxml2 (2.9.8-r1)
(3/4) Installing bind-libs (9.12.3-r0)
(4/4) Installing bind-tools (9.12.3-r0)
Executing busybox-1.28.4-r3.trigger
OK: 188 MiB in 62 packages
/usr/src/app # nslookup users-db
Server: 127.0.0.11
Address: 127.0.0.11#53
Non-authoritative answer:
Name: users-db
Address: 172.20.0.2
I have wasted an entire day on this, and to say I'm not impressed by the unnecessary complexity of what should be a simple task - would be a gross understatement.
Ok, having got that off my chest, I am building a django application using docker-machine, docker-compose, postgresql and redis - by following this tutorial.
I have managed to get the basic tutorial to work - but it does not suit my needs, as I need to create a user and a database for my application - as opposed to using 'postgres' for both.
I have used the answer from #dnephin to a similar question, and modified my code as follows:
I created a new Dockerfile in a new directory ./database/:
FROM postgres:9.6
COPY . /fixtures
WORKDIR /fixtures
RUN /fixtures/setup.sh
./database/setup.sh contents:
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
/etc/init.d/postgresql start
su - postgres # makes no effing difference ...
psql -f create_fixtures.sql
/etc/init.d/postgresql stop
./database/create_fixtures.sql contents:
CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
CREATE USER webuser ENCRYPTED PASSWORD 'deadbeefsnaf0' NOSUPERUSER NOCREATEDB NOCREATEROLE;
GRANT ALL PRIVILEGES ON mydatabase TO webuser;
and finally my postgres service in the docker_compose.yml is modified to use build:
postgres:
build: ./database/
...
When I run docker-compose build, the build goes through the motions and then barfs at where I'm importing the SQL fixtures file via psql:
frothing#themouth:~/path/to/directory$ docker-compose build
redis uses an image, skipping
Building postgres
Step 1/4 : FROM postgres:9.6
---> ff0943ecbb3c
Step 2/4 : COPY . /fixtures
---> fae19dc88da8
Removing intermediate container 84b860aee55c
Step 3/4 : WORKDIR /fixtures
---> aa88438dc69f
Removing intermediate container b801ddc3b374
Step 4/4 : RUN /fixtures/setup.sh
---> Running in ca3e89ec2460
Creating new cluster 9.6/main ...
config /etc/postgresql/9.6/main
data /var/lib/postgresql/9.6/main
locale en_US.utf8
socket /var/run/postgresql
port 5432
Starting PostgreSQL 9.6 database server: main.
psql: FATAL: role "root" does not exist
ERROR: Service 'postgres' failed to build: The command '/bin/sh -c /fixtures/setup.sh' returned a non-zero code: 2
I tried to solve this using the useless documentation on docker for postgresql service - but got no where.
How can I solve this?
Volumes are not available at build time. You can create /var/lib/postgresql/data in your script but it will be overwritten by VOLUME /var/lib/postgresql/data from postgres:9.6 image.
In your case: just use the following docker file:
FROM postgres:9.6
COPY ./create_fixtures.sql /docker-entrypoint-initdb.d/create_fixtures.sql
They get automatically executed once the container starts. Here is an example:
$ docker run -d --name mydb -p 33306:3306 yourtag
$ docker exec -ti mydb psql -U postgres
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------+----------+----------+------------+------------+-----------------------
mydatabase | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | webuser=CTc/postgres
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Outdated answer:
Your script should work on a container except in the fixture you have to execute psql like this:
su postgres -c "psql -f create_fixtures.sql"
su --login postgres does not work because postgres can't open a bash or shell. You can try around with docker run --rm -ti postgres:9.6 bash.
Sorry I have to tell you there is one more error in your sql script: GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser - the keyword DATABASE is necessary here.
Here is a complete log how I tested and can confirm this works:
docker run --rm -ti postgres:9.6 bash
root#be03ab1eb704:/# cat > test.sql <<EOF
> CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
> CREATE USER webuser ENCRYPTED PASSWORD 'asdf123' NOSUPERUSER NOCREATEDB NOCREATEROLE;
> GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser;
> EOF
root#be03ab1eb704:/# pg_createcluster 9.6 main --start
Creating new PostgreSQL cluster 9.6/main ...
/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/9.6/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/9.6/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/9.6/bin/pg_ctl -D /var/lib/postgresql/9.6/main -l logfile start
Ver Cluster Port Status Owner Data directory Log file
9.6 main 5432 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
root#be03ab1eb704:/# /etc/init.d/postgresql start
[ ok ] Starting PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# su postgres -c "psql -f test.sql"
CREATE DATABASE
CREATE ROLE
GRANT
root#be03ab1eb704:/# /etc/init.d/postgresql stop
[ ok ] Stopping PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# exit
exit
The official postgresql docker image automatically imports scripts at the first start of a container. So if you mount your directory with your init sql script to container path '/docker-entrypoint-initdb.d/' it should be run.
For example if you have your import script myImport.sql and it's on your host in a directory /opt/import, you can mount the import directory on container start to your postgres image to /docker-entrypoint-initdb.d and the sql file will be executed after initial database setup.
docker run -p $toHostParam:5432 -e POSTGRES_PASSWORD="$dbPwd" \
-e POSTGRES_USER="$dbUser" \
-e POSTGRES_DB="$dbName" \
-e PGDATA=/opt/pgdata \
-v ${dataDir}:/opt/pgdata \
# look here
# mount of local import dir
-v /opt/import:/docker-entrypoint-initdb.d \
postgres:9.6
Take a look at the postgesql image start script here (from line 126):
https://github.com/docker-library/postgres/blob/master/9.6/docker-entrypoint.sh
If you want a specific db user or database you can also customize your postgresql container with environment variables.
Take a look at the 'Environment Variables' section here: https://hub.docker.com/_/postgres/
Try with this setup.sh
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
su postgres sh -c "/etc/init.d/postgresql start && psql -f create_fixtures.sql && /etc/init.d/postgresql stop"
Try a explicit user when running psql:
psql -U postgres -f create_fixtures.sql
It's the first time that I can't find the answer about some tech problems
Here's my problems:
>> conn=psycopg2.connect(database="mydb", user="postgres", password="123",port=5432)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
My postgreSQL is running
My listeningport is 5432 for sure
root#lanston-laptop:~# psql -l
Password:
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
---------------+----------+----------+------------+------------+-----------------------
checkdatabase | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
mydb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
Thanks a lot!
Your libpq, which is used by psycopg2 expects Postgres socket to be in /var/run/postgresql/ but when you install Postgres from source it is by default it in /tmp/.
Check if there is a file /tmp/.s.PGSQL.5432 instead of /var/run/postgresql/.s.PGSQL.5432. Try:
conn=psycopg2.connect(
database="mydb",
user="postgres",
host="/tmp/",
password="123"
)
Only this solved my problem,
make a symbolic link to the /tmp/.s.PGSQL.5432:
sudo ln -s /tmp/.s.PGSQL.5432 /var/run/postgresql/.s.PGSQL.5432
Thanks to, Sukhjit Singh Sehra - s-postgresql-server-is-running
I originally intended to make it a comment to Tometzky's answer, but well, I have a lot to say here... Regarding the case where you don't call psycopg2.connect directly, but use third-party software.
tl;dr
Set unix_socket_directories in postgresql.conf to /var/run/postgresql, /tmp, and restart PostgreSQL.
intro
I tried PostgreSQL 9.2 (CentOS 7) and 9.5 (Ubuntu Xenial) from distro repos, PostgreSQL 9.3, 9.4, 9.5, 9.6, 10 on CentOS 7 from PostgreSQL repo, PostgreSQL 9.6, 10 on Ubuntu Xenial from PostgreSQL repo. Among them only 9.3 listens to only /tmp:
$ systemctl stop postgresql-9.4 && systemctl start postgresql-9.3
$ lsof -aUp $(ps --ppid 1 -o pid= -o comm= | awk '$2 == "postgres" || $2 == "postmaster" {print $1}')
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 25455 postgres 4u unix 0xffff9acb23bc5000 0t0 6813995 /tmp/.s.PGSQL.5432
$ systemctl stop postgresql-9.3 && systemctl start postgresql-9.4
$ lsof -aUp $(ps --ppid 1 -o pid= -o comm= | awk '$2 == "postgres" || $2 == "postmaster" {print $1}')
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 26663 postgres 4u unix 0xffff9ac8c5474c00 0t0 7086508 /var/run/postgresql/.s.PGSQL.5432
postgres 26663 postgres 5u unix 0xffff9ac8c5477c00 0t0 7086510 /tmp/.s.PGSQL.5432
python-psycopg2
That's not a big deal with psql, just a matter of running the matching binary. But if you, for instance, have python-psycopg2 installed from CentOS's base or update repo. It links dynamically to libpq that OS provides. With 9.3 and 9.4 installed OS provides 9.4's version:
$ alternatives --display pgsql-ld-conf
pgsql-ld-conf - status is auto.
link currently points to /usr/pgsql-10/share/postgresql-9.4-libs.conf
/usr/pgsql-9.3/share/postgresql-9.3-libs.conf - priority 930
/usr/pgsql-9.4/share/postgresql-9.4-libs.conf - priority 940
Current `best' version is /usr/pgsql-9.4/share/postgresql-9.4-libs.conf.
$ ls -l /etc/ld.so.conf.d
lrwxrwxrwx 1 root root 31 Feb 7 02:25 postgresql-pgdg-libs.conf -> /etc/alternatives/pgsql-ld-conf
$ ls -l /etc/alternatives/pgsql-ld-conf
lrwxrwxrwx 1 root root 43 Feb 7 02:25 /etc/alternatives/pgsql-ld-conf -> /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
$ cat /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
/usr/pgsql-9.4/lib/
But libpq that comes with PostgreSQL 9.4 looks for socket in /var/run/postgresql as opposed to 9.3:
$ strings /usr/pgsql-9.3/lib/libpq.so.5 | egrep '/(tmp|var)'
/tmp
$ strings /usr/pgsql-9.4/lib/libpq.so.5 | egrep '/(tmp|var)'
/var/run/postgresql
The solution comes from postinstall scripts of corresponding packages:
$ yum reinstall --downloadonly postgresql94-libs
$ rpm -qp /var/cache/yum/x86_64/7/pgdg94/packages/postgresql94-libs-9.4.15-1PGDG.rhel7.x86_64.rpm --scripts
postinstall scriptlet (using /bin/sh):
/usr/sbin/update-alternatives --install /etc/ld.so.conf.d/postgresql-pgdg-libs.conf pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf 940
/sbin/ldconfig
# Drop alternatives entries for common binaries and man files
postuninstall scriptlet (using /bin/sh):
if [ "$1" -eq 0 ]
then
/usr/sbin/update-alternatives --remove pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
/sbin/ldconfig
fi
Temporarily remove 9.4's alternative:
$ alternatives --remove pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
$ ldconfig
When finished either reinstall postgresql94-libs, or add the alternative back:
$ alternatives --install /etc/ld.so.conf.d/postgresql-pgdg-libs.conf pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf 940
$ ldconfig
pip
If you install psycopg2 with pip on the other hand, it by default installs precompiled package which comes with its own libpq, which looks for socket in /var/run/postgresql:
$ python3.5 -m venv 1
$ . ./1/bin/activate
(1) $ pip install psycopg2
(1) $ python
>>> import psycopg2
>>>Ctrl-Z
[1]+ Stopped python
(1) $ pgrep python
26311
(1) $ grep libpq /proc/26311/maps | head -n 1
7f100b8cb000-7f100b90e000 r-xp 00000000 08:04 112980 /home/yuri/1/lib/python3.5/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
(1) $ strings /home/yuri/1/lib/python3.5/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10 | egrep '/(tmp|var)'
/var/run
/var/run/postgresql
The solution is to ask pip to not install precompiled package, and make pg_config of the proper version of PostgreSQL available:
$ PATH=/usr/pgsql-9.3/lib:$PATH pip install --no-binary psycopg2 psycopg2
You can even add --no-binary switch to requirements.txt:
psycopg2==2.7.3.2 --no-binary psycopg2
unix_socket_directories
The easier option though is to make use of unix_socket_directories option:
Try change port to 5433 instead of 5432
a few years later, using the EnterpriseDB 'graphical' install on OSX 10.8, and pip install of psycopg2 (after linking the /Library/...dylib's as described here) i had this same issue.
for me the correct connect command was conn = psycopg2.connect('dbname=DBNAME user=postgres password=PWHERE host=/tmp/')
In my case with a conda installation had to: sudo ln -s /var/run/postgresql/.s.PGSQL.5432 /tmp/.s.PGSQL.5432
Having this happen to me after a brew upgrade, I googled for brew .s.PGSQL.5432.
Per the suggestion in this answer I ran the following:
postgres -D /usr/local/var/postgres
And got:
2019-10-29 17:43:30.860 IST [78091] FATAL: database files are incompatible with server
2019-10-29 17:43:30.860 IST [78091] DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.5.
I googled that FATAL error and per the suggestion in this answer I ran:
brew postgresql-upgrade-database
That solved it for me.
put vpc_access_connector:
name: project//locations/us-central1/connectors/
and host :'/cloudsql/::
It should work for private Ip postgresql on gcp
Try this once
cd /etc/postgresql/13/main
vi pg_hba.conf
Change the line after this
Database administrative login by Unix domain socket
local all postgres peer
To
local all postgres md5
Then execute following commands
sudo systemctl stop postgresql
sudo systemctl start postgresql
Then run the python program then it will work properly