Assume we need to run second postgres instance in our production environment. First of them (postgres-one) already running and has few databases & data in them. Now I want to update my docker-compose.yaml file and add configuration for the second (postgres-two). Moreover I have to grab some databases info from postgres-one and copy it to postgres-two. Here how I'm trying to achieve this:
docker-compose.yaml
postgres-two:
image: postgres:12.5
depends_on:
postgres-one:
condition: service_started
...
ports:
- "5433:5432"
command: bash -c "chmod +x /usr/local/bin/init.sh && /usr/local/bin/init.sh"
volumes:
- ./data/postgres-two/init-db/init.sh:/usr/local/bin/init.sh
init.sh
#!/bin/bash
# allows you to skip the password prompt for pg_dump
echo "postgres-one:5432:dbname1_one:dbuser1_one:dbpass1_one" > ~/.pgpass
echo "postgres-one:5432:dbname2_one:dbuser2_one:dbpass2_one" >> ~/.pgpass
chmod 600 ~/.pgpass
# gets the data from external database & copies it to internal
pg_dump -h postgres-one -U dbuser1_one dbname1_one | psql -h localhost -U dbuser1_two -d dbname1_two
pg_dump -h postgres-one -U dbuser2_one dbname2_one| psql -h localhost -U dbuser2_two -d dbname2_two
But when I run this I get the error:
psql: error: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
I already tried it w/o -h localhost before, it gave me this as I remember:
psql: error: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
If I add find /var/run/postgresql/ -name .s.PGSQL.5432 in the beginning of the init.sh it'll show nothing. So, as I understand I can't proceed further with psql because postgres server is not running at the moment. And of course I can't run it with postgres / pg_ctl commands because they can't be executed by root:
"root" execution of the PostgreSQL server is not permitted.
And of course docker containers run as the root user by default, and if I change the user it also give me errors such as:
chmod: changing permissions of '/usr/local/bin/init.sh': Operation not permitted
Am I doing something wrong? Or maybe I can get the dumps and apply them in other ways.. somehow?
I think server is failing to start because of volume mapping,
You can refer to this yml,
https://github.com/khezen/compose-postgres/blob/master/docker-compose.yml
Nvm, I solved the problem by placing the init.sh script into the docker-entrypoint-initdb.d folder. But then I also had an access issue with pg_dump because for some reason .pgpass didn't work in my case. For those, who will face the same problem, I fixed it with PGPASSWORD env like that:
PGPASSWORD="dbpass1_one" pg_dump -h postgres-one -U dbuser1_one dbname1_one | psql -h localhost -U dbuser1_two -d dbname1_two
Related
Instead of using /usr/local/pgsql/data, I create my database cluster with the following code
initdb -D /tmp/psql
pg_ctl -D /tmp/psql -l logfile -o "--unix_socket_directories='$PWD'" start
But when I run psql, I get the following error.
psql: error: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
I have read the doc and I cannot find related options for changing the location for database cluster. What have I missed?
I ran into the same issue and specifying the socket directory with the -h option fixes the issue.
psql -h $PWD -p 5432 postgres
I'm trying to extend the postgres Docker image to potentially (via an environment variable flag) execute flyway DB migrations on DB init. My Dockerfile is here:
FROM postgres:9.6
# Install curl and java (for Flyway)
RUN set -x \
&& apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl openjdk-8-jre
# Install Flyway
ENV FLYWAY_VERSION 4.2.0
ENV FLYWAY_INSTALL_DIR /usr/src/flyway
ENV FLYWAY_CONF ${FLYWAY_INSTALL_DIR}/flyway-${FLYWAY_VERSION}/conf/flyway.conf
ENV FLYWAY_EXE ${FLYWAY_INSTALL_DIR}/flyway-${FLYWAY_VERSION}/flyway
RUN mkdir -p ${FLYWAY_INSTALL_DIR} && \
curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/${FLYWAY_VERSION}/flyway-commandline-${FLYWAY_VERSION}.tar.gz | \
tar -xzC ${FLYWAY_INSTALL_DIR} && \
chmod +x ${FLYWAY_EXE}
# Copy migration scripts
ENV MIGRATIONS_LOCATION /flyway/migrations
COPY migrations $MIGRATIONS_LOCATION
COPY init_db.sh /docker-entrypoint-initdb.d/init_db.sh
With my init_db.sh startup script:
#!/bin/bash
set -e
RUN_MIGRATIONS="${RUN_MIGRATIONS:-false}"
DB_URL="jdbc:postgresql://localhost:5432/$DB_NAME"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
CREATE DATABASE $DB_NAME;
EOSQL
if [ "$RUN_MIGRATIONS" == "true" ]; then
echo "running migrations ..."
${FLYWAY_EXE} -user=$POSTGRES_USER -password=$POSTGRES_PASSWORD -url=$DB_URL -locations="filesystem:$MIGRATIONS_LOCATION" migrate
fi
However, when running the container with RUN_MIGRATIONS=true, flyway fails to connect to postgres:
docker build . -t postgres-flyway && docker run -e DB_NAME=db -e RUN_MIGRATIONS=true -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres postgres-flyway
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....LOG: database system was shut down at 2018-08-06 02:19:32 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
done
server started
ALTER ROLE
/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/init_db.sh
CREATE DATABASE
running migrations ...
Flyway 4.2.0 by Boxfuse
ERROR:
Unable to obtain Jdbc connection from DataSource (jdbc:postgresql://localhost:5432/db) for user 'postgres': Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08001
Error Code : 0
Message : Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
The postgres image runs postgres on port 5432 (as usual) so I'm at a loss on why flyway is unable to connect to postgres over localhost:5432.
I also noticed that within this context, pg_isready states that postgres is accepting connections but when specifying the hostname as localhost or 127.0.0.1 it is unable to reach postgres either. That is, by inserting a few pg_isready commands in my init_db.sh script:
...
pg_isready
pg_isready -p 5432
pg_isready -h localhost -p 5432
...
I see the following log output on postgres init:
...
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
localhost:5432 - no response
...
I'm suspicious that I've reached a limitation of postgres' initialize context, but I would like to understand why postgres is unreachable over localhost/127.0.0.1:5432 at this point of initialization.
I had the same problem running flyway when creating a docker image for my database based on the postgres:10.5 image. I added the following to my entrypoint.sh before running flyway, to confirm that the problem I was seeing was caused by the docker-entrypoint.sh change #Nick Maraston posted in his answer:
echo "$(date) - waiting for database to start"
while ! pg_isready -h localhost -p 5432 -d $POSTGRES_DB
do
echo "$(date) - waiting for database to start"
sleep 10
done
The result was that the above code looped for ever. I then replaced it with the following code to restart the database listening for TCP/IP connections on localhost:
pg_ctl -D "$PGDATA" -m fast -w stop
pg_ctl -D "$PGDATA" \
-o "-c listen_addresses='localhost'" \
-w start
Rather than restarting the database like this, a cleaner solution would be to use the JDBC -socketFactory option explained here.
I discovered the problem while digging through the images entry point script. A recent change to the image restricts postgres to only listen for connections over a unix domain socket during internal initialization: https://github.com/docker-library/postgres/pull/440
It is true that postgres docker is listening to a unix socket, eg. /var/run/postgresql/.s.PGSQL.5432. But it is not necessary to force the server to switch its listening address. Postgres database URI allows a connection string to point to a unix socket.
Reference: Connect to a database over a unix socket using SQLAlchemy
The example provided:
export DATABASE_URL=postgres://user:password#/dbname?host=/path/to/unix/socket
I was able to omit the host, and decided to use this environment variable in my /docker-entrypoint-initdb.d/*.sh script instead. Note that no string follows the # symbol, nor is there a host query string here. You may need to explicitly define the host depending on your application.
Solution:
export DATABASE_URL="postgres://$POSTGRES_USER:$POSTGRES_PASSWORD#/$POSTGRES_DB"
I have some docker-compose.yml file. In this file is defined db service (postgres:9.6):
db:
image: postgres:9.6
volumes:
- ./test/data/postgresql/:/var/lib/postgresql/data:delegated
- ./test/bootstrap/postgres:/docker-entrypoint-initdb.d:delegated
ports:
- 15432:5432
environment:
- POSTGRES_DB=test
- POSTGRES_USER=test
- PGDATA=/var/lib/postgresql/data/testdata
And some sh script located in ./test/bootstrap/postgres. In this script i try to connect to postgres by psql like that: psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -h localhost -p 5432 (I know that -h localhost -p 5432 is not necessary but i need it).
When my container is up error occured:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
But after this i restart (after first start this script not executed) my db container then inside in container manually run sh script and it executed successfully. It can connect to localhost and port 5432 even if it is first start.
If i connect in script like that psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" there is no connection error.
Can someone explain me this strange behavior? Why i can't connect to localhost:5432 when container is starting but if connect manually from container it connect successfully?
Thanks!
My understanding is that during the initialization of the container the server is listening on Unix-domain socket.
LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
The scripts in the docker-entrypoint-init.d directory are executed while the server runs in this mode. You're trying to connect via TCP/IP by specifying -h localhost.
Try to remove that argument to connect to the server using Unix sockets.
try to add a sleep command in your script to give enougth time to your database to start correctly.
According to docs at hub.docker.com/_/postgres, to add a custom script to run when your container starts, create a shell script like the one below and create a custom Dockerfile where you'll copy the shell script into /docker-entrypoint-initdb.d/
Assuming you save the following script as initdatabase.sh in the same folder as your custom Dockerfile also defined below in example, when you build the image, you'll have created your custom image which runs the following sql statements defined below.
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE USER user;
CREATE DATABASE customdatabase;
GRANT ALL PRIVILEGES ON DATABASE bloodapp TO user;
// add other sql statements here
EOSQL
FROM mdillon/postgis:9.6-alpine
COPY initdatabase.sh /docker-entrypoint-initdb.d/
I have a working Postgres Dockerfile that I modify and unfortunately after applying modifications Postgres container stops working as expected. I'd like to ask your for explanation of what I'm doing wrong.
Working example
Here's the Postgres Dockerfile that works and which I modify:
# Use ubuntu image
FROM ubuntu
# Install database
RUN apt-get update && apt-get install -y postgresql-9.3
# Switch to postgres user.
USER postgres
# Create databse and user with all privileges to the database.
RUN /etc/init.d/postgresql start && \
psql --command "CREATE DATABASE docker;" && \
psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
psql --command "GRANT ALL PRIVILEGES ON DATABASE docker TO docker;"
# Allow remote connections to the database.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
# Add VOLUMEs to allow backup of config, logs and databases
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
I build it like that:
docker build --tag postgres-image .
Then I create a container:
docker run -d -it -p 32768:5432 --name=postgres postgres-image
And I connect with database:
psql -h localhost -p 32768 -d docker -U docker --password
First modification
I don't need to have any volumes because I'm going to use data-only container that will store all Postgres data. When I remove the line:
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
and do all steps like in working example I get the following error after passing password in the last step:
psql: FATAL: the database system is starting up
FATAL: the database system is starting up
So the question is: Why do I need VOLUME instruction in the Dockerfile?
Second modification
This modification doesn't include the first one. Both modification are independent.
The parameters used in CMD instraction points to default Postgres data directory and configuration file so I wanted to simplify it by setting CMD to the command I always use to start Posgres:
service postgres start
After setting CMD to:
CMD ["service", "postgres", "start]
and doing all steps like in working example I get the following error after passing password in the last step:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 32768?
The question is: Why the command that works on my host system doesn't work in Docker container?
I'm not sure about the first problem. It may be that Postgres doesn't like running on top of the UFS.
The second problem is just that a container will exit when its main process ends. So the command "service postgres start" runs, starts Postgres in the background then immediately exits and the container halts. The first version works because Postgres stays running in the foreground.
But why are you doing this? Why not just use the official Postgres image?
While trying to connect to postgres running locally on my workstation, I get:
$ sudo -u postgres psql -c "create role ..."
could not change directory to "/home/esauer/workspace/cfme"
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
My postgres-server install creates sockets in /var/run/postgresql.
How do I get the client to look in the proper location?
Check the --host option with psql --help.
Then you can make it permanent by setting it in your .psqlrc user file.
In your case try:
psql -h /var/run/postgresql -d your_database