How to properly set VOLUME and CMD instructions in Postgres Dockerfile? - postgresql

I have a working Postgres Dockerfile that I modify and unfortunately after applying modifications Postgres container stops working as expected. I'd like to ask your for explanation of what I'm doing wrong.
Working example
Here's the Postgres Dockerfile that works and which I modify:
# Use ubuntu image
FROM ubuntu
# Install database
RUN apt-get update && apt-get install -y postgresql-9.3
# Switch to postgres user.
USER postgres
# Create databse and user with all privileges to the database.
RUN /etc/init.d/postgresql start && \
psql --command "CREATE DATABASE docker;" && \
psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
psql --command "GRANT ALL PRIVILEGES ON DATABASE docker TO docker;"
# Allow remote connections to the database.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
# Add VOLUMEs to allow backup of config, logs and databases
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
I build it like that:
docker build --tag postgres-image .
Then I create a container:
docker run -d -it -p 32768:5432 --name=postgres postgres-image
And I connect with database:
psql -h localhost -p 32768 -d docker -U docker --password
First modification
I don't need to have any volumes because I'm going to use data-only container that will store all Postgres data. When I remove the line:
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
and do all steps like in working example I get the following error after passing password in the last step:
psql: FATAL: the database system is starting up
FATAL: the database system is starting up
So the question is: Why do I need VOLUME instruction in the Dockerfile?
Second modification
This modification doesn't include the first one. Both modification are independent.
The parameters used in CMD instraction points to default Postgres data directory and configuration file so I wanted to simplify it by setting CMD to the command I always use to start Posgres:
service postgres start
After setting CMD to:
CMD ["service", "postgres", "start]
and doing all steps like in working example I get the following error after passing password in the last step:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 32768?
The question is: Why the command that works on my host system doesn't work in Docker container?

I'm not sure about the first problem. It may be that Postgres doesn't like running on top of the UFS.
The second problem is just that a container will exit when its main process ends. So the command "service postgres start" runs, starts Postgres in the background then immediately exits and the container halts. The first version works because Postgres stays running in the foreground.
But why are you doing this? Why not just use the official Postgres image?

Related

How to fix: 'Psql: could not connect to server: Connection refused' when using Postgres in a docker container in gitlab-ci

I’m currently trying to run fossology in Gitlab CI. Fossology requires an external database that can be set up from a schema created using pg_dump. When I'm trying to use psql I get the title error.
At the moment, I have a script that sets up a container that runs the required version of postgres (9.6). It then tries to run an .sql script via psql in the postgres container via docker exec. Upon doing so it gets the title error.
I have tried specifying both a port and a host when issuing the psql statement, neither of which worked. I have tried using localhost, 127.0.0.1, the IP address of the postgres container and the name of the container as a host. I have tried rewriting things in different scripts, but nothing seems to work.
After extensive google searching, many people seem to have the same error message but not for the same reasons and not usually when using a docker container to host the database.
When I have run the contents of my script in the command line, i do not get this error, the script works fine and I can connect to Fossology. The issue only arises when trying to do the same in Gitlab CI.
The sequence of steps (i.e. pasted line by line) that works when using the command line on Mac:
# creates blank database and hosts it in a docker container
docker run -d --name fossdb -p 5432:5432 postgres:9.6
docker cp /fossology_db_schema.sql fossdb:/fossy.sql
docker exec -it fossdb bash
psql postgres -U postgres
# creates user needed for database to work with fossology
create user fossy with password 'fossy';
create database fossology;
grant all privileges on database fossology to fossy;
\q
# builds the fossology database in the hosted blank database
psql fossology < fossy.sql
psql postgres -U postgres
\connect fossology
exit
What I am attempting in GitLab CI:
# creates container with postgres image
docker run -d --name fossdb -p 5432:5432 --network foss-net postgres:9.6
# creates blank database (error occurs here)
docker exec fossdb psql -h $(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' fossdb) -f ./createBlank.sql -U postgres
# builds fossology database from schema
docker exec fossdb psql -h $(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' fossdb) fossology < ./schema.sql -U postgres
createBlank.sql:
create user fossy with password 'fossy';
create database fossology;
grant all privileges on database fossology to fossy;
Expected results: runs createBlank.sql to create a blank database called fossology, then builds fossology database from schema
Actual results: psql: could not connect to server: Connection refused
Is the server running on host "172.19.0.2" and accepting
TCP/IP connections on port 5432?
Are you sure you set up postgres completely?
A few quick checks you can perform:
(Excuse me, you DID do that. goto suggestion 2)
suggestion 1: Did you tell postgres there is a user with a password? (createuser command)
https://www.postgresql.org/docs/9.2/app-createuser.html
suggestion 2: Did you tell postgres that user can connect, and how? (tcp or local sockets)
https://www.postgresql.org/docs/9.2/auth-pg-hba-conf.html

Accessing Postgres Docker DB through PGAdmin over local internet

So I've built and run a postgres container from a dockerfile on a ubuntu server remotely. The container seems to be up and runing and I can access the db from psql on ubuntu. But I want to be able to connect to the DB remotely via PGadmin over the company network, so far im stuck in having to do it. I'm very new to both docker and Linux, I would really appreciate some help.
Heres a result of the docker ps:
Here is the code for the dockerfile:
FROM base.img:latest
RUN apt-get install postgresql postgresql-contrib -y
# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres
# Copy the dump of table to container
ADD history.dump.2015 /tmp/
# Create a PostgreSQL role named ``docker`` with ``docker`` as the password and
# then create a database `docker` owned by the ``docker`` role.
# Note: here we use ``&&\`` to run commands one after the other - the ``\``
# allows the RUN command to span multiple lines.
RUN /etc/init.d/postgresql start &&\
# psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" && createdb -O docker docker
psql --command "ALTER USER postgres WITH PASSWORD 'abcdefg';"
# Adjust PostgreSQL configuration so that remote connections to the
# database are possible.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.5/main/pg_hba.conf
# And add ``listen_addresses`` to ``/etc/postgresql/9.5/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.5/main/postgresql.conf
# Expose the PostgreSQL port
EXPOSE 5432
# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.5/bin/postgres", "-D", "/var/lib/postgresql/9.5/main", "-c", "config_file=/etc/postgresql/9.5/main/postgresql.conf"]

How to create Postgres backups from docker container

I have a Postgres 9.5.4 database running inside a docker container. I am using the official Postgres images and I have been trying to create backups from my database, but not luck so far. According docker documentation I was expecting the following command to store the dump.tar file in the /releases/ folder in the host machine (outside the temporal docker container)
docker run -i --rm --link my_postgres:postgres --volume /releases/:/tmp/ postgres:latest pg_dumpall > /tmp/dump.tar
But this throws the following error:
pg_dumpall: could not connect to database "template1": could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Any idea what could be wrong?
Actually I have found the solution
docker run -i --rm --link my_container:postgres -v /releases/:/tmp/ postgres:latest bash -c 'exec pg_dumpall -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres > /tmp/my_backup.tar'
Where Docker arguments:
-i For interactive processes, like a shell, bash, etc.
--rm For deleting the container once the backup is finished
--link For linking this temporal container to my_container, which contains the currently running Postgres database.
-v /releases/:/tmp/ Creates shared volume. The content inside the folder /tmp/ in the container (In this case my_backup.tar) will be automatically visible on folder /releases/ in the host machine.
And bash arguments:
pg_dumpall To export PostgreSQL cluster into a script file.
-h "$POSTGRES_PORT_5432_TCP_ADDR" Obtains the ip address out of the Postgres variable.
-p "$POSTGRES_PORT_5432_TCP_PORT" Obtains the Port out of the Postgres variable (These variables are defined only in the temporal container we just created, as result of the linkage with my_container)
-U The username for connecting to the database

Can't add schema for postgres inside docker

I'm trying to build a debian image in Docker that contains nginx, postgresql and php-fpm. I've managed to get nginx and php-fpm working. Postgres is also working but I can't add the schema to the database I have created.
The code from Dockerfile relating to postgres (got it from docker website) is the following:
# Add database
RUN apt-get update && apt-get install -y postgresql-9.4 postgresql-client-9.4 postgresql-contrib-9.4
# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.4`` package when it was ``apt-get installed``
USER postgres
# Create a PostgreSQL role named ``use_name`` with ``user_password`` as the password and
# then create a database `database_name` owned by the ``use_name`` role.
# Note: here we use ``&&\`` to run commands one after the other - the ``\``
# allows the RUN command to span multiple lines.
RUN /etc/init.d/postgresql start &&\
psql --command "CREATE USER user_name WITH SUPERUSER PASSWORD 'user_password';" &&\
createdb -O user_name database_name
# Adjust PostgreSQL configuration so that remote connections to the
# database are possible.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.4/main/pg_hba.conf
# And add ``listen_addresses`` to ``/etc/postgresql/9.4/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.4/main/postgresql.conf
# Reload postgres configuration
RUN /etc/init.d/postgresql stop && /etc/init.d/postgresql start
# Add database schema
COPY ./postgresql/database_name.sql /tmp/database_name.sql
RUN psql -U use_name -d database_name -a -f /tmp/database_name.sql
The error I get is
psql: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Is there another way to do this that I have not seen? Do I need to do something more
The reason you can't connect to the server is because it isn't running. Each line in a Dockerfile is processed in a new container, so any old processes are no longer running, but changes to the filesystem will persist (provided they weren't made to a volume).
You could start the database in the same RUN instruction as the psql command, but normally this sort of thing would be done in an ENTRYPOINT or CMD script that runs when the container is started.
By the far the best plan is to do as #h3nrik suggests and use the official postgres image. Even if you don't want to do this, it's worth looking at the Dockerfile and scripts (e.g https://github.com/docker-library/postgres/tree/master/9.5) used to build the official image to understand how they tackled the same problems.
I would not put the nginx and postgres installations in one single docker container. I would create a separate postgres container and link to it from the nginx/php-fpm container.
The postgres container I would base on the official one. There it is also described how you can add your custom schema to the postgres installation (please have a look at the comment of justfalter, there).

Postgres on Docker. How can I create a database and a user?

I'm trying to create simple postgres server with docker. I use the official postgres image as a base for my container.
My Dockerfile contains these commands:
FROM postgres
USER postgres
RUN /etc/init.d/postgresql start &&\
psql --command "CREATE USER user WITH SUPERUSER PASSWORD 'user';" &&\
createdb -O user app
And when I try to run it I have an error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
What I'm doing wrong?
It's possible that it takes some time for postgres to start accepting connections. The way you've written it, it will call CREATE USER immediately after the start function returns. Try putting a sleep in there and see if it's still a problem.
Use pg_ctl with the -w flag so the command will finish when the server has started. No more wondering about whether we have waited long enough. And we can actually stop the server the same way
sudo -u postgres pg_ctl start -w -D ${PGDATA}
sudo -u postgres psql --command "CREATE USER user WITH SUPERUSER PASSWORD 'user';" &&\
sudo -u postgres createdb -O user app
sudo -u postgres pg_ctl stop -w
Had the same problem inside a script in entry point. Following is my Dockerfile excerpt
# init execution
ENTRYPOINT ["/usr/local/sbin/initpostgres.sh"]
with following commands inside the intipostgres.sh script
su postgres -c "pg_ctl start -l /var/lib/postgresql/logpostgres"
su postgres -c "createuser -s $OPENERPUSER"
adding sleep 1 before the createuser command per #seanmcl suggested correction worked for me :
su postgres -c "pg_ctl start -l /var/lib/postgresql/logpostgres"
sleep 1
su postgres -c "createuser -s $OPENERPUSER"
The problem seems to be that the postgres unix socket is not on your host machine. You can fix this by running the following command.
docker run -p 5432:5432 --volume="/run/postgresql:/run/postgresql" -d --name postgres postgres
The essential part is the --volume flag. It links the folder that includes the unix socket file .s.PGSQL.5432 to the host machine in order to be read by other processes.
This seems like a duplicate question of Installing PostgreSQL within a docker container, is that right OP?