PostgrSQL custom Docker Container - postgresql

I am trying to write a docker entrypoint script, that’s able to start postgreSQL and creating a new named database.
I feel like there is a much better way, than mine. I am happy for every input.
Issues:
a few times I have to restart the container befor it creates the databas and does not crash. Some times I don't know why it workes. I can't recreate it right now.
Freequently I get messages like [unknown]#[unknown] LOG: incomplete startup packet but there shouldn't be any.
Reason for custom docker image: I need PostgreSQL to use ssl for every connection. The official image is sadly not able to allow ssl with real certificates.
Dockerfile:
FROM debian:10.9
RUN apt update && apt install -y \
postgresql postgresql-contrib \
openssl \
net-tools netcat \
vim
RUN mkdir database_stuff
COPY --chown=postgres:postgres ./entrypoint.sh /database_stuff/entrypoint.sh
RUN chmod 777 /database_stuff/entrypoint.sh
COPY --chown=postgres:postgres ./certificates/cert.pem /database_stuff/cert.pem
RUN chmod 600 /database_stuff/cert.pem
COPY --chown=postgres:postgres ./certificates/key.pem /database_stuff/key.pem
RUN chmod 600 /database_stuff/key.pem
COPY --chown=postgres:postgres ./configuration/postgresql.conf /etc/postgresql/11/main/postgresql.conf
COPY --chown=postgres:postgres ./configuration/pg_hba.conf /etc/postgresql/11/main/pg_hba.conf
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
USER postgres
CMD ["/usr/lib/postgresql/11/bin/postgres", "-D", "/var/lib/postgresql/11/main/", "-c", "config_file=/etc/postgresql/11/main/postgresql.conf"]
ENTRYPOINT ["/database_stuff/entrypoint.sh"]
entrypoint.sh script
#!/usr/bin/env bash
echo "START entrypoint.sh"
echo "start postgrSQL"
/usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main/ -c config_file=/etc/postgresql/11/main/postgresql.conf &
until nc -z db 5432; do
sleep 0.1
done
echo "create Database"
psql -U postgres -c 'create database part_screening;'
psql -U postgres -c "create user alstom with encrypted password 'Laser765_f';"
echo "END entrypoint.sh .... Stay open"
tail -f /dev/null
just a log with the unknown#unknown issue
START entrypoint.sh
start postgrSQL
create Database
CREATE DATABASE
CREATE ROLE
END entrypoint.sh .... Stay open
2021-08-02 17:08:52.280 UTC [7] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-02 17:08:52.280 UTC [7] LOG: listening on IPv6 address "::", port 5432
2021-08-02 17:08:52.284 UTC [7] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-02 17:08:52.300 UTC [10] LOG: database system was shut down at 2021-08-02 17:06:42 UTC
2021-08-02 17:08:52.305 UTC [7] LOG: database system is ready to accept connections
2021-08-02 17:08:52.330 UTC [18] [unknown]#[unknown] LOG: incomplete startup packet
2021-08-02 17:08:53.145 UTC [38] [unknown]#[unknown] LOG: incomplete startup packet

Related

Unable to setup Postgres in Nix-Shell

I'm working on a Phoenix/Elixir project which is managed using nix-shell. I'm having issues regarding postgres setup.
I'm starting server using pg_ctl -D $PGDATA start and it outputs as:
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2021-09-01 19:19:00.780 PKT [72367] LOG: listening on IPv4 address "127.0.0.1", port 5432
2021-09-01 19:19:00.784 PKT [72367] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-09-01 19:19:00.806 PKT [72369] LOG: database system was interrupted; last known up at 2021-09-01 19:14:57 PKT
2021-09-01 19:19:00.848 PKT [72369] LOG: database system was not properly shut down; automatic recovery in progress
2021-09-01 19:19:00.849 PKT [72369] LOG: redo starts at 0/16D84A0
2021-09-01 19:19:00.849 PKT [72369] LOG: invalid record length at 0/16D8580: wanted 24, got 0
2021-09-01 19:19:00.849 PKT [72369] LOG: redo done at 0/16D8548
2021-09-01 19:19:00.862 PKT [72367] LOG: database system is ready to accept connections
done
server started
when I execute createuser postgres --createdb --echo in nix-shell envoirnment it produces following error even though the run directory exists.
createuser: could not connect to database template1: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/run/postgresql/.s.PGSQL.5432"?
No other postgres instance is running.
In pg_hba.conf auth method is set to trust.
unix_socket_directories is set to /tmp.
shell.nix looks something like
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
inherit (lib) optional optionals;
elixir = beam.packages.erlangR22.elixir_1_10;
postgresql = postgresql_11;
in
mkShell {
buildInputs = [
ps
elixir
coreutils
which
git
postgresql
redis
doxygen
mongodb-tools
redis-dump
(python37.withPackages(ps: with ps; [ credstash awscli ]))
cmake
nix-prefetch-git
zlib
jq
teleport
]);
# Fix GLIBC Locale
LOCALE_ARCHIVE = lib.optionalString stdenv.isLinux
"${pkgs.glibcLocales}/lib/locale/locale-archive";
LANG = "en_US.UTF-8";
# Put the PostgreSQL and Redis databases in the project diretory.
PGDATA="./.db/postgres";
RDDATA="./.db/redis";
shellHook = ''
ERL_INCLUDE_PATH="${erlangR22}/lib/erlang/usr/include";
}
I totally a noob to Nix and have no idea how to get it working.
Any help would be appreciated, thanks.
You have to tell createuser to connect via the socket in /tmp:
createuser -h /tmp -U postgres --createdb --echo newuser
You've set a custom path in the postgres config, so you probably also need to specify that with pqsl / createuser. In other words createuser -h /tmp/ ....

I can't connect to my postgres database on Docker from Intellij

I'm doing the same steps as described in: https://medium.com/better-programming/connect-from-local-machine-to-postgresql-docker-container-f785f00461a7
but when I try to connect to my postgres (with password mysecretpassword) from Intellij I get the following error:
The specified database user/password combination is rejected: [28P01] FATAL: password authentication failed for user "postgres"
Of course I can connect to my db from cmd command:
$ psql -h localhost -p 5432 -U postgres -W Password for user postgres:
psql (9.5.5, server 10.3 (Debian 10.3-1.pgdg90+1)) WARNING: psql major version 9.5, server major version 10. Some psql features might not work.
Type "help" for help. postgres=# \l
My container is up:
What is going on? I don't have any idea... I use postgres driver 42.2.5 in Intellij
Logs from container:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2020-09-30 11:17:42.613 UTC [45] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-09-30 11:17:42.618 UTC [45] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-09-30 11:17:42.642 UTC [46] LOG: database system was shut down at 2020-09-30 11:17:42 UTC
2020-09-30 11:17:42.649 UTC [45] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2020-09-30 11:17:42.700 UTC [45] LOG: received fast shutdown request
waiting for server to shut down....2020-09-30 11:17:42.705 UTC [45] LOG: aborting any active transactions
2020-09-30 11:17:42.708 UTC [45] LOG: background worker "logical replication launcher" (PID 52) exited with exit code 1
2020-09-30 11:17:42.708 UTC [47] LOG: shutting down
2020-09-30 11:17:42.737 UTC [45] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2020-09-30 11:17:42.836 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-09-30 11:17:42.836 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-09-30 11:17:42.836 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-09-30 11:17:42.845 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-09-30 11:17:42.881 UTC [54] LOG: database system was shut down at 2020-09-30 11:17:42 UTC
2020-09-30 11:17:42.888 UTC [1] LOG: database system is ready to accept connections
create dir for volume: C:\Users{user}\docker\postgres_data\data
Connect to Postgres on Windows:
docker run -p 5432:5432 --name postgres -v C:\Users\{user}\docker\postgres_data\data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=root -d postgres:11.3
Connect to Postgres from CMD and create DB (only after first creation container on empty volume):
docker ps
get the output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e626f4c7b958 postgres:11.3 "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:5432->5432/tcp postgres
Run:
docker exec -it {CONTAINER ID} bash
psql -h localhost -U postgres
psql -U postgres
Connect from Intellij with PASSWORD=root:
It seems that you have running postgresql instance on 5432 port. Check what is running on port 5432:
sudo lsof -i :5432
If there are any records, you could manually kill the processes that are using this port (man kill) and re-run docker container. In my case there was another local instance of postgresql running on this port, so the following helps me:
sudo pkill -u postgres
Or (more preferable) just specify a different port mapping for your docker postgresql (for e.g. 5433):
docker run -p 5433:5432 ...other_your_flags... postgres:latest
try to remove your PostgreSQL the one you have installed locally, if it exists. Because my problem was precisely that the idea did not understand where to connect, to the docker survey or to that installed database on my computer

Running postgres container getting superuser password error?

I am trying to set up a postgres container to start and run initializing the creation of a table. I've succeeded with the straight image from docker but now that I am trying to extend the image a little to create tables when it's produced and I can't get it running. Based off what I've read here How to create User/Database in script for Docker Postgres, this is what I have:
Dockerfile:
FROM library/postgres
COPY init.sql /docker-entrypoint-initdb.d/
init.sql:
CREATE TABLE incident_disposition (
incident_disposition_code VARCHAR,
incident_disposition_code_description VARCHAR
);
From what I understand, FROM library . . . pulls the postgres image from docker hub and the COPY pushes my init.sql script into the entry point so there is no need for a big dockerfile correct?
I then build the image no issue:
Build
docker build -t my_postgres_image .
But when I run I get the issues:
Run
docker run --name testing my_postgres_image --publish 8000:8080 --detach -e POSTGRES_PASSWORD=postgres -d postgres
Errors from logs
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is *not* recommended.
See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
Attempt from comment:
docker container logs testing
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2020-03-26 14:06:51.064 UTC [46] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.072 UTC [46] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.108 UTC [47] LOG: database system was shut down at 2020-03-26 14:06:50 UTC
2020-03-26 14:06:51.119 UTC [46] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE TABLE
2020-03-26 14:06:51.231 UTC [46] LOG: received fast shutdown request
waiting for server to shut down....2020-03-26 14:06:51.232 UTC [46] LOG: aborting any active transactions
2020-03-26 14:06:51.233 UTC [46] LOG: background worker "logical replication launcher" (PID 53) exited with exit code 1
2020-03-26 14:06:51.234 UTC [48] LOG: shutting down
2020-03-26 14:06:51.290 UTC [46] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2020-03-26 14:06:51.345 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-03-26 14:06:51.361 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.387 UTC [64] LOG: database system was shut down at 2020-03-26 14:06:51 UTC
2020-03-26 14:06:51.398 UTC [1] LOG: database system is ready to accept connections
2020-03-26 14:07:27.715 UTC [72] ERROR: relation "incident_disposition" does not exist at character 15
2020-03-26 14:07:27.715 UTC [72] STATEMENT: select * from incident_disposition;
In addition to comments
Due to recent docker image's updates postgres images do not allow to connect to DB without a password from anywhere. So you need to specify username/password
docker run -p 8000:8080 -e POSTGRES_PASSWORD=postgres --name testing -d my_postgres_image
Or if you still don't want to use password, you can just set POSTGRES_HOST_AUTH_METHOD=trust environment variable:
docker run -p 8000:8080 -e POSTGRES_HOST_AUTH_METHOD=trust --name testing -d my_postgres_image
It is a typical Initialization scripts issue.
You can file the full explaination in postgresql docker page. https://hub.docker.com/_/postgres
Here is the brief intro:
1. One common problem is that if one of your /docker-entrypoint-initdb.d scripts fails (which will cause the entrypoint script to exit) and your orchestrator restarts the container with the already initialized data directory, it will not continue on with your scripts.
note:
in your case, you may need clean the historical docker containers(stopped) by
step 1: docker ps |grep
step 2: docker rm -f -v
Or if you are using docker-compose, the historical orchestrator could be easily removed by docker-compose down -v.

Couldn't restore postgres v11 from pg_basebackup

I'm making a backup from my postgresql database (v11) using pg_basebackup like this:
pg_basebackup -h localhost -p 5432 -U postgres -D /tmp/backup -Ft -z -Xs -P
The backup process is completed just fine without any errors.Then as a test I'd like to recover from the backup and for that I do the following steps:
Shutdown the postgres instance.
Delete everything from /var/lib/postgresql/data folder
rm -rf /var/lib/postgresl/data/*
Untar the base archive there:
tar xvf /tmp/backup/base.tar.gz -C /var/lib/postgresql/data
Untar the wal archives to a temporary directory
tar xvf /tmp/backup/pg_wal.tar.gz -C /tmp/archived_wals
Create a file recovery.conf in the /var/lib/postgresql/data folder:
touch /var/lib/postgresql/data/recovery.conf
chown postgres:postgres /var/lib/postgresql/data/recovery.conf
Specify the recovery command in the recovery.conf file:
restore_command = 'cp /path/archived_wals/%f "%p" '
Restart the postgres instance, and wait for the recovery to be finished.
However when postgres starts up I get the following error in the logs:
2020-02-04 11:34:52.599 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-02-04 11:34:52.599 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-02-04 11:34:52.613 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-02-04 11:34:52.709 UTC [18] LOG: database system was interrupted; last known up at 2020-02-04 08:39:54 UTC
cp: can't stat '/tmp/archived_wals/00000002.history': No such file or directory
2020-02-04 11:34:52.735 UTC [18] LOG: starting archive recovery
2020-02-04 11:34:52.752 UTC [18] LOG: invalid checkpoint record
2020-02-04 11:34:52.752 UTC [18] FATAL: could not locate required checkpoint record
2020-02-04 11:34:52.752 UTC [18] HINT: If you are not restoring from a backup, try removing the file "/var/lib/postgresql/data/backup_label".
2020-02-04 11:34:52.753 UTC [1] LOG: startup process (PID 18) exited with exit code 1
2020-02-04 11:34:52.753 UTC [1] LOG: aborting startup due to startup process failure
2020-02-04 11:34:52.769 UTC [1] LOG: database system is shut down
I guess I'm missing something, maybe I need to do additional configuration to postgres to be able to restore a pg_basebackup backup?
Try this:
/pgsql-11/bin/pg_basebackup -D /pgdata/pg11/data -X stream --waldir=/pglog/pg11/wal_log/ -P -c fast -h remoteserver\localserver -U username
Let me know if that works.
After all, it was my bad.
I enclosed the destination placeholder within % instead of double quotes in recovery.conf.
However the way I described for the backup and restore is valid and since I corrected the typo it works just fine for me, so I leave the question as it is for further reference.

I need to build docker image with postgres with data included [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
So, I need docker image with: my program that use postgresql, postgresql and some data in it
I use this article: https://medium.com/#sharmaNK/build-postgres-docker-image-with-data-included-489bd58a1f9e
My steps:
docker-compose build //Creates ps_image_with_data
docker run --name ps-export -p 5432:5432 -i postgres
Connect to 5432 db, using dbeaver; create table; put some rows in the table
docker exec -it my_container_with_data_id bash
mkdir /postgres
cp -r /var/lib/postgresql/data/* /postgres
docker commit ps-export ps_image_with_data
docker run --env PGDATA=postgres -p 5432:5432 -i ps_image_with_data // here I've tried both ps_image_with_data and new image that was created on step 7
And after 8 step I got this:
2018-11-08 12:44:52.549 UTC [1] LOG: listening on IPv4 address
"0.0.0.0", port 5432 2018-11-08 12:44:52.549 UTC [1] LOG: listening
on IPv6 address "::", port 5432 2018-11-08 12:44:52.615 UTC [1] LOG:
listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2018-11-08 12:44:52.750 UTC [22] LOG: database system was
interrupted; last known up at 2018-11-08 11:56:27 UTC 2018-11-08
12:45:25.042 UTC [22] LOG: could not remove cache file
"global/pg_internal.init": Permission denied 2018-11-08 12:45:25.042
UTC [22] LOG: could not remove cache file
"base/13067/pg_internal.init": Permission denied 2018-11-08
12:45:25.042 UTC [22] LOG: database system was not properly shut
down; automatic recovery in progress 2018-11-08 12:45:25.127 UTC [22]
LOG: redo starts at 0/166CF68 2018-11-08 12:45:25.127 UTC [22] LOG:
invalid record length at 0/166D048: wanted 24, got 0 2018-11-08
12:45:25.127 UTC [22] LOG: redo done at 0/166D010 2018-11-08
12:45:25.259 UTC [22] PANIC: could not rename file
"pg_logical/replorigin_checkpoint.tmp" to
"pg_logical/replorigin_checkpoint": Permission denied 2018-11-08
12:45:25.357 UTC [1] LOG: startup process (PID 22) was terminated by
signal 6: Aborted 2018-11-08 12:45:25.358 UTC [1] LOG: aborting
startup due to startup process failure 2018-11-08 12:45:25.359 UTC [1]
LOG: database system is shut down
Can someone help me to sort it out?
How about
docker run --name ps-export -p 5432:5432 -i postgres
Connect to 5432 db, using dbeaver; create table; put some rows in the table
Make a dump (data.sql) to your host
Add in your docker-compose.yml, under the psql server
`
volumes:
- ./data.sql:/docker-entrypoint-initdb.d/init.sql
docker-compose up
If you want to include data.sql in the image, instead of step 4, copy it to /docker-entrypoint-initdb.d/init.sql in your dockerfile.