run psql on postgres dockerfile - postgresql

From a postgres standard images I install postgis and after I should run psql command to CREATE EXTENSION POSTGIS. I think that my problem is to set the correct -h parameter in the psql command but I'm not sure about it. Any idea?
FROM postgres:12.1
MAINTAINER xxx
#ARG A_DB_USER='postgres'
#ARG A_DB_PASS='postgres'
ARG A_DB_NAME='postgres'
ARG A_TZ='Europe/Zurich'
#ENV DB_USER=${A_DB_USER}
#ENV DB_PASS=${A_DB_PASS}
ENV DB_NAME=${A_DB_NAME}
ENV TZ=${A_TZ}
# Adjusting Timezone in the system
RUN echo $TZ > /etc/timezone && \
apt-get update && apt-get install -y tzdata && \
rm /etc/localtime && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
# install postgis
RUN apt-get update && \
apt-get install -y postgis && \
apt-get clean
USER postgres
#Add password "postgres" to user postgres, create db, add .sql
RUN /etc/init.d/postgresql start && psql -U postgres -d mydb -h localhost -c "CREATE EXTENSION postgis;"
EXPOSE 5432
The error is: psql: error: could not connect to server: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
ERROR: Service 'istsos-db' failed to build: The command '/bin/sh -c /etc/init.d/postgresql start && psql -U postgres -d istsos -h localhost -c "CREATE EXTENSION postgis;"' returned a non-zero code: 2
EDIT:
I run it with a docker-compose
version: '3.7'
services:
app-db:
build:
context: ./
dockerfile: Dockerfile-pg
image: app-pg:1.0.0
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app
POSTGRES_DB: app-db
volumes:
- ./docker-entrypoint-initdb.d/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh
- v-app-pgdata:/var/lib/postgresql
- v-app-pglog:/data/var/log/postgresql
- v-app-pgconf:/etc/postgresql
app-main:
build:
context: ./
dockerfile: Dockerfile-tar-cp
image: app-main:1.0.0
restart: always
ports:
- 80:80
volumes:
v-app-pgdata:
name: v-app-pgdata
v-app-pglog:
name: v-app-pglog
v-app-pgconf:
name: v-app-pgconf
I tried to use POSTGRES_DB: app-db and to specify psql -h app-db
But docker-build returns:
No PostgreSQL clusters exist; see "man pg_createcluster" ... (warning).
psql: error: could not connect to server: could not translate host name "app-db" to address: Name or service not known
ERROR: Service 'app-db' failed to build: The command '/bin/sh -c /etc/init.d/postgresql start && psql -U postgres -d app -h app-db -c "CREATE EXTENSION postgis;"' returned a non-zero code: 2
I also try with localhost without success.
Any ideas?

Try the below code for your run script.
RUN /etc/init.d/postgresql start &&\
psql --command "CREATE EXTENSION postgis;"
EDIT:
Please add the below to make sure ports are open and pg listen to external IP's as well.
RUN echo "host all all 0.0.0.0/0 md5" >> /folder/to/conf/pg_hba.conf
add listen to all ip's
RUN echo "listen_addresses='*'" >> /folder/to/conf/postgresql.conf

Related

Error when connecting to postgres from a Python container

I'm trying to set up two communicating containers as different services
PostgreSQL for holding the data
Python with Jupyter notebook for querying the database
Here's my Dockerfile
FROM python:3.8.8-slim-buster
RUN mkdir /project
WORKDIR /project
COPY . /project/
RUN apt-get update && \
apt-get install --no-install-recommends -y \
build-essential \
bash
RUN pip install --upgrade --no-use-pep517 \
pip \
setuptools \
wheel \
jupyterlab \
ipython-sql \
psycopg2-binary
EXPOSE 8888
and my docker-compose.yml
version: '3'
services:
db:
image: postgres:12.1-alpine
environment:
- POSTGRES_DB=ps_db
- POSTGRES_USER=ps_user
- POSTGRES_PASSWORD=ps_pass
ports:
- "5432:5432"
web:
build:
context: .
command:
jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root
volumes:
- .:/project
ports:
- "8888:8888"
depends_on:
- db
Although I'm pretty sure my settings are fine, whenever I run
import psycopg2
conn = psycopg2.connect(database="ps_db", user="ps_user", password="ps_pass", host="0.0.0.0", port=5432)
I get this error
OperationalError: could not connect to server: Connection refused
Is the server running on host "0.0.0.0" and accepting
TCP/IP connections on port 5432?
You need to set the 0.0.0.0 IP to 'db', because the postgres runs in another container called 'db', this is the name of your service. So your conn will look like this:
conn = psycopg2.connect(database="ps_db", user="ps_user", password="ps_pass", host="db", port=5432)

Rename volume in docker compose, with docker derived from postgres

I have this docker-compose
version: '3.7'
services:
app-db:
build:
context: ./
dockerfile: Dockerfile-pg
image: app-pg:1.0.0
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app
volumes:
- ./docker-entrypoint-initdb.d/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh
- v-app-pgdata:/var/lib/postgresql
- v-app-pglog:/data/var/log/postgresql
- v-app-pgconf:/etc/postgresql
app-main:
build:
context: ./
dockerfile: Dockerfile-tar-cp
image: app-main:1.0.0
restart: always
ports:
- 80:80
volumes:
v-app-pgdata:
name: v-app-pgdata
v-app-pglog:
name: v-app-pglog
v-app-pgconf:
name: v-app-pgconf
so an app container and a postgres derived container:
#docker build -t app-pg:1.0.0 -f Dockerfile-pg .
#docker run -d --name appC-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres app-pg:1.0.0
FROM postgres:12.1
MAINTAINER xxx
#ARG A_DB_USER='postgres'
#ARG A_DB_PASS='postgres'
ARG A_DB_NAME='app'
ARG A_TZ='Europe/Zurich'
#ENV DB_USER=${A_DB_USER}
#ENV DB_PASS=${A_DB_PASS}
ENV DB_NAME=${A_DB_NAME}
ENV TZ=${A_TZ}
# Adjusting Timezone in the system
RUN echo $TZ > /etc/timezone && \
apt-get update && apt-get install -y tzdata && \
rm /etc/localtime && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
# install postgis
RUN apt-get update && \
apt-get install -y postgis && \
apt-get clean
USER postgres
#Add password "postgres" to user postgres, create db, add .sql
#RUN /etc/init.d/postgresql start && \
# psql --command "ALTER USER ${DB_USER} WITH PASSWORD '${DB_PASS}'; SET TIME ZONE '${TZ}';" && \
# createdb -O ${DB_USER} ${DB_NAME} -E UTF8 && \
# psql -d ${DB_NAME} -c 'CREATE EXTENSION postgis'
EXPOSE 5432
My problem is that postgres default dockerfile have this line:
VOLUME /var/lib/postgresql/data
So even if I create a named Volumes with the same folder, my docker-compose create 4 and not 3 volumes, one unnamed due to that line.
How is it possible solve this issue?
I had the same problem as you and I solved it by editing the docker-compose.yml the following way:
Create a volume for the data
volumes:
v-app-pgdata:
In your declaration of the database service, in the volumes clause, you need to change - v-app-pgdata:/var/lib/postgresql to:
- v-app-pgdata:/var/lib/postgresql/data
The problem was that you where not mounting the volume to the correct place, and therefore, it had to create another volume that is declared in the base image VOLUME /var/lib/postgresql/data. If you run this container without attaching a volume to this mount point, it will automatically create it, and with a random name.
However, it has been pointed out to me that if the OP strictly needs a volume in /var/lib/postgresql then this solution wont work.
Hope this helps, as it worked for me.

Docker container can't connect circleCI postgres database

I am trying to set up a circleCI test, I have created a database in circleCI and I have a docker container which needs to connect to the database, but it can't. Inside my docker container is a script which before it does anything it runs pg_isready, this cannot connect to the database. Here's my circle job creation
postgres_tests:
docker:
- image: circleci/python:3.7
- image: circleci/postgres:9.6.2-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_DB: my_test
steps:
- setup_remote_docker:
docker_layer_caching: true
- attach_workspace:
at: /tmp/workspace
- run:
name: Install awscli docker-squash
working_directory: /
command: sudo pip3 install awscli docker-squash
- run: eval `aws ecr get-login --no-include-email --region eu-west-1`
- checkout
- run: echo 'export PATH=/usr/lib/postgresql/9.6/bin/:$PATH' >> $BASH_ENV
- run: sudo apt-get update && sudo apt-get install -y postgresql-client
- run: psql -h localhost -U postgres --command "ALTER USER postgres WITH PASSWORD 'password';"
- run:
name: run_pg_tests
working_directory: /tmp/workspace
command: |
/tmp/workspace/sql/t/run_tests.sh
The run_tests.sh is a script which pulls my docker image from the company repo and then does a docker run on that image.
I have read other people have issues where the database isn't ready so to test this I added pg_isready before the docker run
So my script looks like this
DB_HOST=`psql -X -A -h localhost -U postgres -p 5432 -t -c "select inet_server_addr()"`
DB_PORT=5432
DB_NAME=my_test
DB_USER=postgres
DB_PASSWORD=password
pg_isready -h "${DB_HOST}" -p "${DB_PORT}"
#restore database from supplied image
docker run \
-e SAPIENTIA_DB_HOST=$DB_HOST \
-e SAPIENTIA_DB_PORT=$DB_PORT \
-e SAPIENTIA_DB_NAME=$DB_NAME \
-e SAPIENTIA_DB_PASSWORD=$DB_PASSWORD \
-e SAPIENTIA_DB_USER=$DB_USER \
$EMPTY_DB_FULL_PATH \
path_to_file/file
I have also tried setting the DB_HOST variable directly to 'localhost' the result is exactly the same
Here's what I get as a result:
127.0.0.1:5432 - accepting connections
127.0.0.1:5432 - no response
I have also tried re-running the test with ssh and connecting myself. Same result, I can connect to the database, but i I then run docker exec and try to connect from inside the docker container it can't connect.
I'm pretty stumped here, so any help would be useful.
EDIT: I've found this documentation page about your issue:
It is not possible to start a service in remote docker and ping it directly from a primary container or to start a primary container that can ping a service in remote docker. To solve that, you’ll need to interact with a service from remote docker, as well as through the same container
That line is not 100% clear to me, but I understand that they tell us that we should run the containers we want to communicate from another container manually. Therefore:
- run:
name: run_pg_tests
working_directory: /tmp/workspace
command: |
docker run -d --name postgres --env POSTGRES_USER=postgres --env POSTGRES_DB=my_test circleci/postgres:9.6.2-alpine
/tmp/workspace/sql/t/run_tests.sh
Since the postgres container is not accessible anymore through the local network, your up check could be docker exec postgres pg_isready
You can then set your DB_HOST to postgres in your run script.
Original answer:
I'm not well versed into CircleCI configuration, but my guess would be that your Docker container you run manually is not attached to the same network as the containers launched by CircleCI.
From what I see in the documentation, you can specify the hostname of the service container:
The name the container is reachable by. By default, container services are accessible through localhost
So maybe if you try something lile this:
- image: circleci/postgres:9.6.2-alpine
name: postgres
environment:
POSTGRES_USER: postgres
POSTGRES_DB: my_test
You can then set your DB_HOST to postgres in your run script.

How can I access the postgres.conf of a postgres container on gitlab-ci?

On Gitlab-CI I set up a postgres service for my database and would like to inspect the config file of it.
For this I let postgres return the location of the config file but when I go to that directory, it is empty.
How can I access it?
.gitlab-ci.yaml:
image: maven:3.5.3-jdk-8
services:
- postgres
variables:
POSTGRES_DB: custom_db
POSTGRES_USER: custom_user
POSTGRES_PASSWORD: custom_pass
connect:
image: postgres
script:
- export PGPASSWORD=$POSTGRES_PASSWORD
- psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT 'OK' AS status;"
- psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SHOW config_file;"
- cd /var/lib/postgresql/data
- dir
- ls -a
- cat postgresql.conf
The respective job output:
Running with gitlab-runner 11.8.0 (4745a6f3)
on docker-auto-scale 72989761
Using Docker executor with image postgres ...
Starting service postgres:latest ...
Pulling docker image postgres:latest ...
Using docker image sha256:30bf4f039abe0affe9fe4f07a13b00ea959299510626d650c719fb10c4f41470 for postgres:latest ...
Waiting for services to be up and running...
Pulling docker image postgres ...
Using docker image sha256:30bf4f039abe0affe9fe4f07a13b00ea959299510626d650c719fb10c4f41470 for postgres ...
Running on runner-72989761-project-7829066-concurrent-0 via runner-72989761-srm-1551974294-08e28deb...
Cloning repository...
Cloning into '/builds/kimtang/SpringBootTimeWithSwagger'...
Checking out 1399a232 as master...
Skipping Git submodules setup
$ export PGPASSWORD=$POSTGRES_PASSWORD
$ psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT 'OK' AS status;"
status
--------
OK
(1 row)
$ psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SHOW config_file;"
config_file
------------------------------------------
/var/lib/postgresql/data/postgresql.conf
(1 row)
$ cd /var/lib/postgresql/data
$ dir
$ ls -a
.
..
$ cat postgresql.conf
cat: postgresql.conf: No such file or directory
ERROR: Job failed: exit code 1
Why does it state it is in /var/lib/postgresql/data but then can not be found?
You're connected to a remote docker instance via psql and you're checking a local directory. If you really want to check what's going on on the service docker image then ssh into the worker and then use the docker exec -i -t <container_name> /bin/sh command to log into the container. You will have to make the job run for a long time though so put some sleep in there.

Connect to postgresql container from another container (Docker)

I have am trying to follow this tutorial and set up a postgresql container.
I have the following script:
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$#"
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec $cmd
And the following docker-compose.yml:
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
The problem is that when I run docker-compose up I get the following error:
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
server_1 | Postgres is unavailable - sleeping
server_1 | psql: could not translate host name "192.168.64.2:5432" to address: Name or servi
ce not known
Now I have tried setting the host as database, localhost, 0.0.0.0, and even the containers IP but nothing works, I have no idea what it should be or how to debug it, I am not 100% sure how docker-compose links the containers.
do not use depends_on. try it with "links"
version: '2'
services:
server:
build: .
ports:
- 3030:3030
links:
- database
#environment could be usefull too
environment:
DATABASE_HOST: database
command: ["./setup/wait-for-postgres.sh", "localhost:5432", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
for more informations https://docs.docker.com/compose/compose-file/#links
May be an old thread to answer but I have been using depends_on with the following docker-compose file
version: '3.4'
volumes:
postgres_data:
driver: local
services:
postgres:
image: postgres
volumes:
- ./postgres_data:/var/lib/postgresql:rw
- ./deployments:/opt/jboss/wildfly/standalone/deployments:rw
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
ports:
- 5432:5432
keycloak:
image: jboss/keycloak
environment:
POSTGRES_ADDR: postgres
POSTGRES_DATABASE: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: password
KEYCLOAK_USER: admin
KEYCLOAK_PASSWORD: Pa55w0rd
ports:
- 8080:8080
- 9990:9990
depends_on:
- postgres
The tutorial skips over a few things, and is confusing in that it mentions the wait-for-it.sh script, but then shows a much simplified version that doesn't work if you pass hostname:port as one argument to it.
I had a crack at getting this to work and both for future me and others I will add the steps below. I did this on MacOS, and have both docker and docker-compose installed as well as nodejs.
I don't have your node app handy so I used the one as described here https://nodejs.org/de/docs/guides/nodejs-docker-webapp/
I have the following directory structure:
/src/package.json
/src/server.js
/.pgpass
/docker-compose.yml
/Dockerfile
/wait-for-postgres.sh
The contents of these files is listed below.
Steps
From the ./src directory run $ npm install (creates package-lock.json)
Fix pgpass permissions with $ chmod 600 .pgpass
Make the script executable $ chmod +x wait-for-postgres.sh
From the root directory $ docker-compose up
It will pull the postgres image and build the node app container.
When that's done it will wait for postgres and when postgres is up you'll see it ready.
Files
The src files are exactly as per the node js dockerize link above
/src/package.json
{
"name": "docker_web_app",
"version": "1.0.0",
"description": "Node.js on Docker",
"author": "First Last <first.last#example.com>",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.16.1"
}
}
/src/server.js
'use strict';
const express = require('express');
// Constants
const PORT = 8080;
const HOST = '0.0.0.0';
// App
const app = express();
app.get('/', (req, res) => {
res.send('Hello world\n');
});
app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
.pgpass
This uses the username:password postgres:postgres and is purely for development demo purposes. In the wild you will use some other method of secrets management and never ever commit a pgpass file to version control
#host:port:db:user:pass
db:5432:*:postgres:postgres
docker-compose.yml
I have added the wait-for-postgres.sh script as a managed volume, in the original question it was bundling it in with the app src which was weird.
I have also mounted the .pgpass file in the root user's home directory, which psql will look in for auto-password completion. If you don't have some method of supplying this then you'll get an error:
psql: fe_sendauth: no password supplied
Notice the command for the server container is referring to database which is a valid docker-compose internal dns name for the postgres container.
version: '2'
services:
server:
build: .
ports:
- 3030:3030
depends_on:
- database
volumes:
- ./wait-for-postgres.sh:/usr/app/setup/wait-for-postgres.sh
- ./.pgpass:/Users/root/.pgpass
command: ["/usr/app/setup/wait-for-postgres.sh", "database", "--", "node", "src"]
database:
image: postgres
environment:
- "POSTGRES_USER=postgres"
- "POSTGRES_PASSWORD=postgres"
- "POSTGRES_DB=tide_server"
ports:
- 5432:5432
Dockerfile
I have modified this from the node js tutorial, pinning it to the Debian "buster" version and also installing psql which it needs for that script.
FROM node:10-buster
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
RUN apt-get -y update - && \
apt-get -y install libpq-dev && \
apt-get -y install postgresql-client-11
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm#5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
# Bundle app source
COPY . .
EXPOSE 8080
CMD [ "node", "server.js" ]
wait-for-postgres.sh
I have modified the script very slightly because I ran the "shellcheck" linter and it complained about a few things. I realise this script is from the docker tutorial page.
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$*"
export PGPASSFILE=./pgpass
until psql -h "$host" -U "postgres" -c '\l'; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec "$cmd"
The problem here is the host itself.
psql -h **"$host"** -U "<USER>" -c '\l'
You are passing a wrong HOSTNAME "localhost:5432" / "192.168.64.2:5432"
What I did is setup a ~/.pgpass that has
localhost:5432:DB:USER:PASSWORD
and instead of passing "localhost:5432", omit the port. Just use "localhost"
This works for me ...