create db if not exists in postgres .sh file - postgresql

I want to execute these lines in the below .sh file:
#!/bin/sh +x
sudo su postgres
psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'test_db'" | grep -q 1 || psql -U postgres -c "CREATE DATABASE test_db"
But only 'sudo su postgres' is executing and the 2nd line is not executing. Can someone help me execute those 2 lines

Just pass your psql command to su:
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM pg_database WHERE datname = 'test_db'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE test_db\""
If you want to pass the datname value as a parameter, replace it by $1 in the script ($1, $2, etc. expand as the first, second, etc. parameters you pass to your script):
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM pg_database WHERE datname = '$1'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE $1\""
and call your script like this:
./script.sh test_db
You can even have optional parameters. For instance, if you want an optional parameter for the table name (pg_database in your example):
db=pg_database
if [ -n "$2" ]; then db="$2"; fi
sudo su postgres -c "psql -U postgres -tc \"SELECT 1 FROM $db WHERE datname = '$1'\" | \
grep -q 1 || psql -U postgres -c \"CREATE DATABASE $1\""
and call your script like this:
./script.sh test_db
to use the default, else:
./script.sh test_db other_pg_database

I suggest you update your sudo configuration so you can directly express that you only need to as postgres and not root:
sudo -u postgres ...
For example:
user host = (postgres) command
where user is your user, host is the host name you want this apply (ALL?), and command is the name of your command (ALL?) possible prefixed with "NOPASSWD: " if you don't want to require a password.
Then do the action and deal with the error if needed instead of guarding against it:
sudo -u postgres bash -c "psql -U postgres -c "CREATE DATABASE $1"
Possible single quoting the $1 and if you want that to be robust, escape any single quotes in the database name. I showed you a shell (bash) in the above, so can easily tag on error handling.

Related

bash script to have a postgres DB in a docker container

I'm having trouble in creating a Postgres DB using this bash script:
#! /bin/bash
docker pull postgres
docker run --name coverage-postgres -e POSTGRES_PASSWORD=password -p 5432:5432 -d postgres
export CONTAINER_ID=$(sudo docker ps -a | grep coverage-postgres | head -c12)
sleep 2s
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "create user coverage_user with password 'password';"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "create database coverage owner coverage_user;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "grant all privileges on database coverage to coverage_user;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "\c coverage coverage_user" # it seems useless...
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "CREATE TABLE IF NOT EXISTS postal_codes (id,...;"
sleep 0.5
sudo docker exec -it $CONTAINER_ID psql -U postgres -c "CREATE UNIQUE INDEX ... ;"
# exit from container
exit
# restart container
docker start $CONTAINER_ID
In particular, the database is created, the user is created, the table is created but... it's not in the coverage db but in postgres db.
I've tried to add "CREATE TABLE coverage.postal_codes" but coverage is a db and not a schema and it didn't work.
I've tried to use: psql -U coverage_user but the system tells me that database coverage_user doesn't exist.
So of course I thought "I have to specify the database of course!". Then I've tried to use: psql -U coverage as the name of the database but this time, the system makes fun of me and, changing its mind, tells me that the role coverage doesn't exists.
I tried a workaround: within the command -c "\c coverage coverage_user" I concatenated the other commands this way:
-c "\c coverage coverage_user; CREATE TABLE...; CREATE UNIQUE INDEX...;"
but, of course, neither this worked at all.
I make a premise: I know there are other ways to do this but I would like to understand what I am missing with these specific commands.
Solution
#! /bin/bash
docker pull postgres
docker run --name coverage-postgres -e POSTGRES_PASSWORD=password -p 5432:5432 -d postgres
export CONTAINER_ID=$(sudo docker ps -a | grep coverage-postgres | head -c12)
sleep 2s
docker exec -it $CONTAINER_ID psql -U postgres -c "create user coverage_user with password 'password';"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U postgres -c "create database coverage owner coverage_user;"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U postgres -c "grant all privileges on database coverage to coverage_user;"
sleep 0.5
docker exec -it $CONTAINER_ID psql -U coverage_user -c "CREATE TABLE IF NOT EXISTS postal_codes (id int)" coverage
Explanation
https://www.postgresql.org/docs/9.2/app-psql.html
psql [option...] [dbname [username]]
Just add dbname after options. And change user as -U option. You can pass dbname also as an rgument -d

How to in insert into a specific schema in postgres using psql pipe command

I have using psql pipe command to copy a table from one database to another in Postgres. It is working fine. But I need to copy the table to a specific schema in the new database. I have gone through the documentation (used -n option for specifying schema name) but it is not working.
Command:
pg_dump -U postgres -h localhost -p 1212 -d dbname -t tablename -Ft | pg_restore -U postgres -h localhost -p 1213 -d dbname -n schemaname
you can't do it with pg_dump|pg_restore sequence. you need to alter table t set schema s; in restored db
I do it that this way:
pg_dump -U postgres -h localhost -p 1212 -d dbname -t tablename | sed "sed/oldschemaname/newschemaname/" | psql -U postgres -h localhost -p 1213 -d dbname -n schemaname
With the corresponding regular expression

How to run Postgres in a docker Alpine Linux container?

I'm trying to get Postgresql 10.0 working in a docker container. I have the following Dockerfile:
FROM postgres:10.0-alpine
RUN apk add openrc --no-cache
USER postgres
RUN /etc/init.d/postgresql start
RUN psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user') THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;"
RUN createdb main
EXPOSE 5432
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
CMD ["/usr/lib/postgresql/10.0/bin/postgres", "-D", "/var/lib/postgresql/10.0/main", "-c", "config_file=/etc/postgresql/10.0/main/postgresql.conf"]
I got following error:
/bin/sh: /etc/init.d/postgresql: not found
Seems that /etc/init.d/postgresql is really missing. What am I doing wrong?
I finished with this Dockerfile:
FROM postgres:10.0-alpine
USER postgres
RUN chmod 0700 /var/lib/postgresql/data &&\
initdb /var/lib/postgresql/data &&\
echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf &&\
echo "listen_addresses='*'" >> /var/lib/postgresql/data/postgresql.conf &&\
pg_ctl start &&\
psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'main'" | grep -q 1 || psql -U postgres -c "CREATE DATABASE main" &&\
psql -c "ALTER USER postgres WITH ENCRYPTED PASSWORD 'mysecurepassword';"
EXPOSE 5432
This dockerfile create database 'main' (if not exists), starts postgres and sets default user password
I was able to do it using exec commands, so I can run postgres in a running container, in this case alpine 3.7. Note $CONTAINER_NAME is the container id from docker ps:
# Install postgresql, create user, db & start the daemon (for testing)
sudo docker exec $CONTAINER_NAME sh -c 'apk add postgresql'
sudo docker exec $CONTAINER_NAME sh -c 'addgroup -S postgres && adduser -S postgres -G postgres'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'mkdir -p /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /run/postgresql/'
sudo docker exec $CONTAINER_NAME sh -c 'chmod -R 777 /var/lib/postgresql/data'
sudo docker exec $CONTAINER_NAME sh -c 'chown -R postgres:postgres /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'initdb /var/lib/postgresql/data'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf'
sudo docker exec --user postgres $CONTAINER_NAME sh -c 'pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log'
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'buildpgpass';\""
sudo docker exec --user postgres $CONTAINER_NAME sh -c "psql --command \"CREATE DATABASE builddb;\""
You can also include this in your Dockerfile like so:
# Postgres
RUN apk add postgresql=11.1-r0
RUN (addgroup -S postgres && adduser -S postgres -G postgres || true)
RUN mkdir -p /var/lib/postgresql/data
RUN mkdir -p /run/postgresql/
RUN chown -R postgres:postgres /run/postgresql/
RUN chmod -R 777 /var/lib/postgresql/data
RUN chown -R postgres:postgres /var/lib/postgresql/data
RUN su - postgres -c "initdb /var/lib/postgresql/data"
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
RUN su - postgres -c "pg_ctl start -D /var/lib/postgresql/data -l /var/lib/postgresql/log.log && psql --command \"ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';\" && psql --command \"CREATE DATABASE builddb;\""
This seems to work great if you want to test something out on the fly.
There are multiple problems with your dockerfile. I guess you are trying to start the server when building the image to create the main database. This won't work since each command in the dockefile will execute in its own layer, and thus when you reach RUN psql ... the database won't be started since it was started in a different layer. So you need to group the commands in one line.
Second problem is that the file /etc/init.d/postgresql does not exist. The server can be started using the postgres command:
RUN postgres &\
psql --command "IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'user')\
THEN CREATE USER user WITH SUPERUSER ENCRYPTED PASSWORD 'password'; END IF;" &\
createdb main

How to push a local database dump into a docker container?

I tried:
ssh root#mytutumnode "docker exec -ti mydockercontainerid su postgres -c 'psql mydatabasename'" < mydump.sql
but it executes only
ssh root#mytutumnode "docker exec -ti mydockercontainerid su postgres -c 'psql mydatabasename'"
... and omits < mydump.sql part.
psql --list and alikes do work.
Here's the Makefile i ended up using:
# Convenience cli shortcuts
LOCAL_DOCKER_IP = 192.168.99.100
STAGING_IP = DOCKER_NODE_IP_HERE
RELEASE_IP = IP_FROM_THE_SERVER_WITH_THE_ORIGINAL_DB
DOCKER_DB_ID = $(shell ssh root#$$STAGING_IP "docker ps | grep database" | awk '{print $$1}')
download_staging_db:
psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres -c 'drop database mydb'
psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres -c 'create database mydb'
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'pg_dump mydb'" | psql -h $(LOCAL_DOCKER_IP) -p 5432 -U postgres mydb
upload_release_db_to_staging_db:
ssh root#$(RELEASE_IP) "cd / && su postgres -c 'pg_dump my db'" > temp.sql
scp temp.sql root#$(STAGING_IP):/temp.sql
ssh root#$(STAGING_IP) "docker cp /temp.sql $(DOCKER_DB_ID):/temp.sql"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'drop database mydb'"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'create database mydb'"
ssh root#$(STAGING_IP) "docker exec $(DOCKER_DB_ID) su postgres -c 'psql mydb < /temp.sql'"
rm temp.sql

After restoring dump in postgres some primary keys are missing

After dumping a database using pg_dump like so:
pg_dump --verbose --host=<host> --username=<user> -W --encoding=UTF-8 -j 10 --file=dump_bak --format=d --dbname=<database>
and trying to reimport it with:
pg_restore -d <database> --host=<host> -n public --username=<user> -W --exit-on-error --format=d -j 10 --verbose dump_bak
…we are missing some of our Primary Keys. It looks like a few have been restored, but not all.
Any ideas?
It seems that Postgres is a bit buggy there concerning existing connections. We have found a workaround:
Dump it with:
pg_dump --verbose --host=<dbhost> --username=<username> --encoding=UTF-8 --file=<dumpfile> --format=d --jobs=10 --dbname=<dbname>
Restore it with:
export PGPASSWORD="<pwd>"
#prevent new connections, kill existing connections
sudo -u postgres /usr/bin/psql -q -c "update pg_database set datallowconn = 'false' where datname = '<dbname>'; SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '<dbname>';"
#kick db
sudo -u postgres dropdb <dbname>
#recreate it
sudo -u postgres createdb <dbname>
#allow connections again
sudo -u postgres /usr/bin/psql -q -c "update pg_database set datallowconn = 'true' where datname = '<dbname>';"
#import data from dump
pg_restore --verbose -d <dbname> --host=$1 --username=<username> -j 4 <dumplocation>