Creating Postgresql database with dummy data using docker compose - postgresql

I created a docker compose file for implementing Postgresql database wih pgAdmin.
version: "3.8"
services:
# environnement postgresql
postgresdb:
image: postgres
container_name: postgres
restart: always
user: root
environment:
POSTGRES_USER: root
POSTGRES_PASSWORD: root
APP_DB_USER: docker
APP_DB_PASS: docker
APP_DB_NAME: org_db
networks:
- postgres-network
ports:
- '5432:5432'
expose:
- '5432'
volumes:
- ./data/dbstore/postgres/postgres:/var/lib/postgresql/data
- ./data/setup/postgres:/docker-entrypoint-initdb.d
pgAdmin:
image: dpage/pgadmin4
container_name: pgAdmin
restart: always
user: root
links:
- postgresdb
depends_on:
- postgresdb
networks:
- postgres-network
environment:
PGADMIN_DEFAULT_EMAIL: root#root.com
PGADMIN_DEFAULT_PASSWORD: root
ports:
- "5050:80"
volumes:
- ./data/dbstore/postgres/pgadmin:/var/lib/pgadmin4/storage
- ./data/setup/postgres/pgadmin4.db:/pgadmin4.db
entrypoint: /bin/sh -c "cp /pgadmin4.db /var/lib/pgadmin4/storage/pgadmin4.db && cd /pgadmin4 && /entrypoint.sh"
When running the docker compose up command, the applications are correctly launched. My issue is that at the same time I'm trying to create a database with sample data.
After some researches (a response was found here: https://graspingtech.com/docker-compose-postgresql), I created a sh file:
#!/bin/bash
set -e
export PGPASSWORD=$POSTGRES_PASSWORD;
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE USER $APP_DB_USER WITH PASSWORD '$APP_DB_PASS';
CREATE DATABASE $APP_DB_NAME;
GRANT ALL PRIVILEGES ON DATABASE $APP_DB_NAME TO $APP_DB_USER;
\connect $APP_DB_NAME $APP_DB_USER
BEGIN;
CREATE TABLE IF NOT EXISTS employee (
id serial primary key,
name varchar(250) NOT NULL,
age int NOT NULL,
salary numeric NOT NULL,
PRIMARY KEY (id)
);
INSERT INTO employee(ID,NAME,AGE,SALARY) values (1,'Dave',30,90000),(2,'Peter',28,80000);
COMMIT;
EOSQL
I expepected a database to be created and some data to be inserted. Unfortunatelly, I have 3 issues, first my sh file is not correcly interpreted and database is not created at all.
postgres | server started
postgres | CREATE DATABASE
postgres |
postgres |
postgres | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sh
postgres | /usr/local/bin/docker-entrypoint.sh: /docker-entrypoint-initdb.d/init.sh: /bin/bash^M: bad interpreter: No such file or directory
postgres exited with code 126
postgres |
postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization

Related

How to run an *.sql script for a specific database on init with a postgres docker container using multiple databases?

I'm using a postgres service through docker compose with multiple databases in the same container (one for keycloak and a test db for non-identity managment application data). As per the postgres image docs, I've mounted a volume with a shell script to create multiple databases. This part is working fine, and once the containers are up and running I can access the databases.
Inside the volume there's also an .sql file with a query for creating tables and inserting rows into these tables. While the docker logs show that the tables are created and the insertions are performed, I can't see the tables on either of the created databases. How would I go about ensuring this script runs on the testdb?
Docker compose file:
volumes:
postgres_data:
driver: local
services:
test-webportal-db:
env_file:
- ./.env
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
- ./db-init:/docker-entrypoint-initdb.d
environment:
- POSTGRES_DB=${TEST_DB}
- POSTGRES_MULTIPLE_DATABASES=${POSTGRES_MULTIPLE_DATABASES}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
ports:
- 5432:5432
test-webportal-db-adminer:
image: adminer:latest
restart: always
depends_on:
- test-webportal-db
ports:
- 8090:8080
test-webportal-identity:
env_file:
- ./.env
image: quay.io/keycloak/keycloak:latest
environment:
- DB_VENDOR=${DB_VENDOR}
- DB_ADDR=${DB_ADDR}
- DB_DATABASE=${DB_DATABASE}
- DB_USER=${DB_USER}
- DB_SCHEMA=${DB_SCHEMA}
- DB_PASSWORD=${DB_PASSWORD}
- KEYCLOAK_USER=${KEYCLOAK_USER}
- KEYCLOAK_PASSWORD=${KEYCLOAK_PASSWORD}
ports:
- 8080:8080
depends_on:
- test-webportal-db
Shell script:
#!/bin/bash
set -e
set -u
# function to create user and database with POSTGRES_PASSWORD
function create_user_and_database() {
local database=$1
echo " Creating user and database '$database'"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
CREATE USER $database WITH PASSWORD '$POSTGRES_PASSWORD';
CREATE DATABASE $database;
GRANT ALL PRIVILEGES ON DATABASE $database TO $database;
EOSQL
}
# create a database for each one listed in the POSTGRES_MULTIPLE_DATABASES env variable
if [ -n "$POSTGRES_MULTIPLE_DATABASES" ]; then
echo "Multiple database creation requested: $POSTGRES_MULTIPLE_DATABASES"
for db in $(echo $POSTGRES_MULTIPLE_DATABASES | tr ',' ' '); do
create_user_and_database $db
done
echo "Multiple databases created"
fi
I managed a workaround by removing the .sql file and placing the queries into the shell script as follows:
psql -U $TEST_DB $TEST_DB << EOF
<queries here>
EOF

SQL schema files not running on init in Dockererized PostgreSQL instance

I am trying to create table 'User' in my database with a docker container with PostgreSQL and pgAdmin on docker-compose up.
I am following this StackOveflow article here.
DockerFile
FROM postgres:latest
COPY schema.sql /docker-entrypoint-initdb.d/
docker-compose.yaml
version: "3.8"
services:
db:
build: .
restart: always
environment:
POSTGRES_DB: xxx
POSTGRES_USER: admin
POSTGRES_PASSWORD: xxx
PGDATA: /var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
- db-data:/var/lib/postgresql/data
pgadmin:
image: dpage/pgadmin4:latest
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: admin#xxx.com
PGADMIN_DEFAULT_PASSWORD: xxx
PGADMIN_LISTEN_PORT: 80
ports:
- "8080:80"
volumes:
- pgadmin-data:/var/lib/pgadmin
links:
- "db:pgsql-server"
volumes:
db-data:
pgadmin-data:
schema.sql
create Table Users (
id SERIAL,
uuid VARCHAR(100) NOT NULL
)
The container and admin console starts up fine. However, the table is not generated and cannot be found in pgAdmin.
I exec-ed into the container and the file schema.sql is well copied into /docker-entrypoint-initdb.d/ directory.
In the PostgreSQL Docker repository, the documentation mentions that my data directory has to be empty or the files in the init folder will not be run. Could this be my issue?
How can I create my User table in my dockerized PostgreSQL when I run the container?

How to restore database from dump or sql file in docker using volume?

I am trying to run my database in docker which already have some data into it, But when I run it in docker it gives empty .. So how to restore my database in docker PostgreSQL image.
First, Start your docker posgtres container. Then execute the following command:
docker exec -i <CONTAINER> psql -U <USER> -d <DB-NAME> < <PATH-TO-DUMP>
<CONTAINER> : Container ID or container name
<USER> : User of PostgreSQL DBMS. If you use the standard postgres image with no specific config the user is postgres
<DB-NAME>: The name of the DB. This DB should already exist. You can create a db when running your postgres container by specifying an environment variable named POSTGRES_DB
<PATH-TO-DUMP>: Specify here the path to your dump file
Example:
docker exec -i mypostgres psql -U postgres -d mydb < backup.sql
Hope this helps.
I am getting this error , when I dockerize Pdi Jobs.
ERROR (version 8.3.0.0-371, build 8.3.0.0-371 from 2019-06-11 11.09.08 by buildguy) : An error occurred, processing will be stopped:
2019/10/31 05:02:03 - Table input.0 - Error occurred while trying to connect to the database
2019/10/31 05:02:03 - Table input.0 -
2019/10/31 05:02:03 - Table input.0 - Error connecting to database: (using class org.postgresql.Driver)
2019/10/31 05:02:03 - Table input.0 - Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
2019/10/31 05:02:03 - Table output.0 - ERROR (version 8.3.0.0-371, build 8.3.0.0-371 from 2019-06-11 11.09.08 by buildguy) : An error occurred intialising this step:
My docker-compose File:
version: '3.1'
services:
pdijobs:
image: pdijobs
restart: always
build:
context: .
dockerfile: Dockerfile
volumes:
- ./data/jasper-data:/usr/local/tomcat/webapps
ports:
- "8080:8080"
depends_on:
- datawarehouse-db
environment:
DB_TYPE: postgresql
DB_HOST: datawarehouse-db
DB_NAME: data-warehouse
DB_USER: postgres
DB_PASSWORD: admin
datawarehouse-db:
image: postgres:10
restart: "no"
environment:
POSTGRES_DB: data-warehouse
POSTGRES_USER: postgres
POSTGRES_PASSWORD: admin
volumes:
postgres-data:
pgdata:

create postgres database if it does not exist on docker-compose start up

I have the following docker-compose file:
version: '3.5'
services:
postgres:
container_name: postgres_container
image: postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
PGDATA: /data/postgres
volumes:
- postgres:/data/postgres
onrun:
psql -h=localhost -P=${POSTGRES_PASSWORD} -U=${POSTGRES_USER} -tc "SELECT 1 FROM pg_database WHERE datname = 'premiership'" | grep -q 1 || psql -h=localhost -P=${POSTGRES_PASSWORD}-U=${POSTGRES_USER}c "CREATE DATABASE premiership"
ports:
- "5432:5432"
networks:
- postgres
restart: unless-stopped
The above does not work because there is no onrun in docker-compose services.
All I want to do is create a database if it does not exist but this is insanely difficult because I don't know when the service is up.
Also it is not easy to do this in postgres. I tried mapping a volume so that an initdb.sql is ran:
volumes:
- postgres:/data/postgres
- ./initdb/1_schema.sql:/docker-entrypoint-initdb.d/1_schema.sql
1_schema.sql looks like this:
SELECT datname
FROM pg_database
WHERE datname='premiership'';
IF datname='
premiership'
THEN CREATE DATABASE premiership PASSWORD 'test';
END IF;
THe database is not created and when I run docker-compose logs I don't see anything about running the script.
Have you tried setting POSTGRES_DB variable in your docker-compose.yml environment ?
image: postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
POSTGRES_DB: premiership
PGDATA: /data/postgres
volumes:
see https://hub.docker.com/_/postgres :
POSTGRES_DB
This optional environment variable can be used to define a different
name for the default database that is created when the image is first
started. If it is not specified, then the value of POSTGRES_USER will
be used.

docker faild to restore postgres backup

I'm trying to create and restore postgres backup using docker.
the docker failed to do it and gives me the following error:
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
dockerfile:
FROM postgres:11
ENV POSTGRES_USER postgres
ENV POSTGRES_PASSWORD postgres
ENV POSTGRES_DB dbName
COPY backup.backup /
COPY initdb.sh /docker-entrypoint-initdb.d
initdb.sh:
pg_restore --username=postgres --create --exit-on-error --verbose --dbname=dbName backup.backup
docker-compose.yml:
version: '2'
services:
db:
image: postgres:11
expose:
- "5432"
ports:
- "15432:5432"
volumes:
- dock-volume:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD= postgres
- POSTGRES_DB= dbName
volumes:
dock-volume:
I tried to add the environment variables to docker-compose.yml but it doesnt help..
You should not create a Dockerfile for the Postgres because you already have the definition in your Dockercompose file. The variables that you define under environment are visible inside Postgres. If you want to have a backup and make sure that is running when you initialize you can do:
volumes:
- ~/Downloads/data/my_buckup.psql:/docker-entrypoint-initdb.d/stage.sql
Then when Postgres initialize this script will be run. You can see the documentation here under the section How to extend this image
I hope it makes sense