Why can't I connect to postgres container outside of docker-compose? - postgresql

I am doing (or planning to do) some spatial db work with OpenStreetMaps data using Postgres. I am working on a mac m1, and decided the best way is to run a Postgres db with PostGis installed in a docker container. I can't connect from my host machine, but I can connect from another container added in the same docker-compose.
In order to set the database up for importing OSM data, I need to change some of the config values, as well as install PostGis. So I have written my own dockerfile that completes these tasks on startup. To do so I use a two .sh files. This dockerfile works mostly correctly, as I can connect from another docker container running pgadmin4, but my backend application, azure data studio, and psql from terminal cannot connect, getting could not connect to server: Operation timed out Is the server running on host "172.31.0.2" and accepting TCP/IP connections on port 5432?.
My assumption is that I'm missing or adding something incorrectly when I run the Postgres container but no amount of googling bears fruit! Any help would be amazing.
Docker compose
version: '3.8'
services:
db:
build:
./database
environment:
- POSTGRES_HOST_AUTH_METHOD=trust
ports:
- "5432:5432"
Dockerfile
FROM postgres:14-bullseye
LABEL maintainer="PostGIS Project - https://postgis.net"
ENV POSTGIS_MAJOR 3
ENV POSTGIS_VERSION 3.3.2+dfsg-1.pgdg110+1
RUN apt-get update \
&& apt-cache showpkg postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR \
&& apt-get install -y --no-install-recommends \
# ca-certificates: for accessing remote raster files;
# fix: https://github.com/postgis/docker-postgis/issues/307
ca-certificates \
\
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /docker-entrypoint-initdb.d
COPY ./initdb-postgis.sh /docker-entrypoint-initdb.d/10_postgis.sh
COPY ./update-postgis.sh /usr/local/bin
Essentially a copy of the postgis docker image, but that allows me to install other extensions in the initdb-postgis.sh and update-postgis.sh files, shown below:
initdb-postgis.sh
#!/bin/sh
set -e
export PGUSER="$POSTGRES_USER"
"${psql[#]}" <<- 'EOSQL'
ALTER SYSTEM SET listen_addresses = '*';
ALTER SYSTEM SET wal_level = minimal;
ALTER SYSTEM SET max_wal_senders = 0;
ALTER SYSTEM SET checkpoint_timeout = '1d';
ALTER SYSTEM SET checkpoint_completion_target = 0.90;
ALTER SYSTEM SET shared_buffers = '8GB';
ALTER SYSTEM SET max_wal_size = '10GB';
ALTER SYSTEM SET min_wal_size = '1GB';
CREATE DATABASE template_postgis IS_TEMPLATE true;
EOSQL
for DB in template_postgis "$POSTGRES_DB"; do
echo "Loading PostGIS extensions into $DB"
"${psql[#]}" --dbname="$DB" <<-'EOSQL'
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS postgis_topology;
CREATE EXTENSION IF NOT EXISTS hstore;
EOSQL
done
update-postgis.sh
#!/bin/sh
set -e
export PGUSER="$POSTGRES_USER"
POSTGIS_VERSION="${POSTGIS_VERSION%%+*}"
for DB in template_postgis "$POSTGRES_DB" "${#}"; do
echo "Updating PostGIS extensions '$DB' to $POSTGIS_VERSION"
psql --dbname="$DB" -c "
-- Upgrade PostGIS (includes raster)
CREATE EXTENSION IF NOT EXISTS postgis VERSION '$POSTGIS_VERSION';
ALTER EXTENSION postgis UPDATE TO '$POSTGIS_VERSION';
-- Upgrade Topology
CREATE EXTENSION IF NOT EXISTS postgis_topology VERSION '$POSTGIS_VERSION';
ALTER EXTENSION postgis_topology UPDATE TO '$POSTGIS_VERSION';
-- Upgrade hstore
CREATE EXTENSION IF NOT EXISTS hstore;
"
I'm connecting from the host using the IP given when I docker inspect the container id, so the docker compose network shouldn't be an issue. As I mentioned before, connection works fine from inside the docker compose, with both a pgadmin4 and unix script running osm2pgsql. Both can connect using either the IP from docker inspect, or the name of the service, in this case 'db'.
Whilst I admit I am relitively inexperienced with both docker and postgis, normally some amount of googling helps me find the issue, but no this time - please help!

It turns out that other processes (namely other postgres db's) were running on port 5432. So following this post fixed the issue, and I can now connect using psql.

Related

Postgresql doesn't working with Citus and pg_stat_statements at the same time

So, I built the PostgreSQL with citus extension in docker.
I use the official documentation in citus, then I run this command in the terminal.
docker run -d --network citus-network --name citus_coordinator -p 5500:5432 -e POSTGRES_PASSWORD=mypassword citusdata/citus:11.1
Then Database successfully building.
But I want to create the pg_stat_statements extension.
I configuration the postgresql.conf file.
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all
pg_stat_statements.max = 10000
track_activity_query_size = 2048
Then I restarted the PostgreSQL container. Wrote the this query
SELECT * FROM pg_stat_statements;
in terminal.
I saw this error
[55000] ERROR: pg_stat_statements must be loaded via shared_preload_libraries
I didn't understand, why the config file didn't see this extension, What was my mistake?
If using citus, citus should always be your first shared preload library. So your postgres.conf should look like:
shared_preload_libraries = 'citus,pg_stat_statements'
pg_stat_statements.track = all
pg_stat_statements.max = 10000
track_activity_query_size = 2048
If you have workers, you need to change their postgresql.conf as well.
Then you need to restart the databases (coordinator and workers) which you can achieve by:
docker restart <container_name>
You can confirm this worked with:
SHOW shared_preload_libraries;
Solved
I am running the PostgreSQL container with postgres:latest image. Then I entered the container. So, I installed the citus from here into the container. Then installed apt-get install postgresql-contrib into the container.Next step, run create extension pg_stat_statements; And everything worked for me.

A container is a database server. How to ask it's Dockerfile to complete its construction after that container has started?

I am using a postgis/postgis Docker image to set a database server for my application.
The database server must have a tablespace created, then a database.
Then each time another application will start from another container, it will run a Liquibase script that will update the database schema (create tables, index...) when needed.
On a terminal, to prepare the database container, I'm running these commands :
# Run a naked Postgis container
sudo docker run --name ecoemploi-postgis
-e POSTGRES_PASSWORD=postgres
-d -v /data/comptes-france:/data/comptes-france postgis/postgis
# Send 'bash level' commands to create the directory for the tablespace
sudo docker exec -it ecoemploi-postgis
bin/sh -c 'mkdir /tablespace && chown postgres:postgres /tablespace'
Then to complete my step 1, I have to run SQL statements to create the tablespace in a PostGIS point of view, and create the database by a CREATE DATABASE.
I connect myself, manually, under the psql of my container :
sudo docker exec -it ecoemploi-postgis bin/sh
-c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR"
-p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'
And I run manally these commands :
CREATE TABLESPACE data LOCATION '/tablespace';
CREATE DATABASE comptesfrance TABLESPACE data;
exit
But I would like to have a container created from a single Dockerfile having done all the needed work. The difficulty is that it has to be done in two parts :
One before the container is started. (creating directories, granting them user:group).
One after it is started for the first time : declaring the tablespace and creating the base. If I understand well the base image I took, it should be done after an entrypoint docker-entrypoint.sh has been run ?
What is the good way to write a Dockerfile creating a container having done all these steps ?
The PostGIS image "is based on the official postgres image", so it should be able to use the /docker-entrypoint-initdb.d mechanism. Any files you put in that directory will be run the first time the database container is started. The postgis Dockerfile already uses this directory to install the PostGIS extensions into the default database.
That means you can put your build-time setup directly into the Dockerfile, and copy the startup-time script into that directory.
FROM postgis/postgis:12-3.0
RUN mkdir /tablespace && chown postgres:postgres /tablespace
COPY createdb.sql /docker-entrypoint-initdb.d/20-createdb.sql
# Use default ENTRYPOINT/CMD from base image
For the particular setup you describe, this may not be necessary. Each database runs in an isolated filesystem space and starts with an empty data directory, so there's not a specific need to create an alternate data directory; Docker style is to just run multiple databases if you need isolated storage. Similarly, the base postgres image will create a database for you at first start (named by the POSTGRES_DB environment variable).
In order to run a container, your Dockerfile must be functional and completed.
you must enter the queries in a bash file and in the last line you have to enter an ENTRYPOINT with this bash script

Artifactory upgrade fail, postgres 9.5 -> 9.6 upgrade instructions needed

I had planned an upgrade of artifactory from 6.7.5 to 6.8.1. As part of the upgrade I checked jfrog's repo on github and it looks like they have a new recommended nginx and postgres version.
The current docker-compose is using postgres 9.5 and the new default version if 9.6. Simply pulling down postgres 9.6 however does not do an inplace upgrade.
FATAL: database files are incompatible with server DETAIL: The data
directory was initialized by PostgreSQL version 9.5, which is not
compatible with this version 9.6.11.
The upgrade instructions do not mention anything about how to do the upgrade.
The examples provided in github (https://github.com/jfrog/artifactory-docker-examples) are just examples.
Using them in production could cause issues and backwards compatibility is not guaranteed.
To get over the PostgreSQL matter when upgrading, I would suggest:
$ docker-compose -f yml-file-name.yml stop
edit the yml-file-name.yml and change the docker.bintray.io/postgres:9.6.11 to docker.bintray.io/postgres:9.5.2
$ docker-compose -f yml-file-name.yml up -d
Artifactory should be upgraded after following this, however it will keep using the previous version of the PostgreSQL DB
I have been able to upgrade database using following approach:
Dump all database to an SQL script using old database image; store it in a volume for future import:
# Override PostgreSQL image used to export using old binaries
printf "version: '2.1'\nservices:\n postgresql:\n image: docker.bintray.io/postgres:9.5.2\n" > image_override.yml
started_container=$(docker-compose -f artifactory-pro.yml -f image_override.yml run -d -v sql_dump_volume:/tmp/dump --no-deps postgresql)
# Dump database to a text file in a volume (to make it available for import)
docker exec "${started_container}" bash -c "until pg_isready -q; do sleep 1; done"
docker exec "${started_container}" bash -c "pg_dumpall --clean --if-exists --username=\${POSTGRES_USER} > /tmp/dump/dump.sql"
docker stop "${started_container}"
docker rm --force "${started_container}"
Back up old database directory and prepare a new one:
mv -fv /data/postgresql /data/postgresql.old
mkdir -p /data/postgresql
chown --reference=/data/postgresql.old /data/postgresql
chmod --reference=/data/postgresql.old /data/postgresql
Run a new database image with mounting dump script from step 1. It processes SQL scripts upon startup when setting up a new database, provided it's started as postgres something. We just don't need to leave the server running afterwards, so I provided --version to make entrypoint execute, import the data and quit:
docker-compose -f artifactory-pro.yml run --rm --no-deps -e POSTGRES_DB=postgres -e POSTGRES_USER=root -v sql_dump_volume:/docker-entrypoint-initdb.d postgresql postgres --version
After all this is done, I was able to start Artifactory normally with docker-compose -f artifactory-pro.yml up -d and it started up normally, applying rest of schema and file upgrade procedure as usual.
I have also prepared a script that basically does the above steps along with some additional checks and cleanup. Feel free to use if you find it useful.

docker - addressing environment variables used in parent image

I need to create plpythonu extension on the postgres:10.3 image. For this, I extended the image with an installation command:
FROM postgres:10.3
RUN apt update && apt install -y postgresql-plpython-10
Now I need to run a psql command create extension plpythonu;. But at this point the postgres server is not running yet, so when running docker build . I get an error:
Step 3/3 : RUN psql -c "create extension plpythonu;";
---> Running in 037066c120ea
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
How do I fit this command in? And is there a way to do this without copypasting the entire docker-entrypoint.sh from the original postgres image? Maybe there's a way to specify this in docker-compose?
When step-2 is over you Postgres database down, that is why in Step-3 database is not running.
You can combine step-2 and step-3 together.
RUN apt update && \
apt install -y postgresql-plpython-10 && \
psql -c "create extension plpythonu;"
It means && (logical and) if previous commands successfully over then only execute next one.

How to upgrade Cloudera Manager Postgres database

I have Cloudera Manager 5.9 installed on Ubuntu 12.04 with embedded postgres database. I upgraded Ubuntu to 14.04 using do-release-upgrade. In the process, Postgres also got upgraded from 8.4 to 9.3. Now when I try to start the CM database via:
# sudo service cloudera-scm-server-db start
I get the following error in CM db.log:
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 8.4, which is not compatible with this version 9.3.15.
How do I get past this? I have looked at a lot documentation which talks about dumping the postgres database via pg_dump and restoring via psql, but I don't know how this applies in the context of cloudera manager, especially when the database is not coming up.
On Ubuntu 12.04 when everything is working, I believe the dump can be taken like this:
#pg_dump -h localhost -p 7432 -U scm > /tmp/scm_server_db_backup.$(date +%Y%m%d)
I can try to create an empty database and restore the dump to this one using psql. But how do I configure cdh to point to this database?
As you suggest, you need to find a way to "convert" 8.4 data files to 9.3 data files.
Using pg_dump will need a working PostgreSQL 8.4 instance. So, basically you need a working Postgresql 8.4 (think VM or Docker), then copy your existing 8.4 fiels to that VM/Docker, have that VM/Docker provide a plain-text dump [plain SQL, so compatible with any version), restore that plain-text dump to your 9.3 instance).
You can try :
Create a VirtualMachine or a Docker instance with Postgresql 8.4
deployed.
Locate the main data directory (usually /var/lib/postgresql/8.4/main on Ubuntu, but this might differ) of
your upgraded Cloudera machine. Backup this directory and keep in
safe.
Stop PostgreSQL on your VM/Docker if necessary. Locate the main data directory (usually /var/lib/postgresql/8.4/main on Ubuntu, but
this might differ).
Replace the previous found directory by a copy of your existing 8.4/main content (the one on your upgraded machine having now PG 9.3) to your VM/Docker.
Restart PostgreSQL 8.4 on the VM/Docker
Use pg_dumpall to create a full backup :
pg_dumpall > dump.sql
Transfer dump.sql to your Cloudera machine, and restore it. You might need to drop previous schemas/databases :
psql -f dump.sql postgres
I am able to resolve this problem using the following process:
Step 1: Take a dump of the running postgres database on Ubuntu 14.02
# sudo su
# su - postgres
# pg_dump -h localhost -p 7432 -U scm scm > scm.sql
Step 2: Upgrade Ubuntu to 16.04
# sudo do-release-upgrade
...
Step 3: Rename the old data directory
# mv /var/lib/cloudera-scm-server-db/data/ /var/lib/cloudera-scm-server-db/data9-3
Step 4: Restart cloudera-scm-server-db service. This will create an empty database which we will populate using the backup taken in step 1
# sudo service cloudera-scm-server-db restart
Step 5: Now restore the database
# sudo su
# su - postgres
# psql -h localhost -p 7432 -U scm
(password can be obtained like this: grep password /etc/cloudera-scm-server/db.properties)
scm> \i scm.sql
Step 6: Now restart cloudera-scm-server service:
# sudo service cloudera-scm-service restart