I am following the Pushkin Quickstart guide.
At pushkin init site, I get an error that no container was found for test_db_1:
...
Pulling test_db (postgres:11)...
11: Pulling from library/postgres
Digest: sha256:8e096175da9b7a1d5f073e4ff0b2058a68b3110dc9c26bcee0975d25ad1c008e
Status: Downloaded newer image for postgres:11
Creating pushkin_test_db_1 ... done
Creating pushkin_message-queue_1 ... done
Creating pushkin_api_1 ... done
Creating pushkin_server_1 ... done
Starting test_db ... done
Creating local test database
No container found for test_db_1
Failed to run create database command in test_db container: 1
If I open Docker Desktop > pushkin > pushkin_test_db_1 > Logs, I see
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the superuser.
For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all connections without a password.
This is *not* recommended. See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
I tried opening ./pushkin.yaml, adding a password there, and running pushkin init site but nothing has changed. My guess is that the command represented by Creating pushkin_test_db_1 doesn't provide Docker a password to the database. Looking on the postgres github, this behavior seems to be a new "feature."
Does anyone have a recommendation to fix this issue?
I think I found a fix. First, I deleted all of the old docker containers to prevent conflicts when trying to create new ones. Then, from a new directory, I ran pushkin site default to initialize the filesystem. Next, in ./pushkin/docker-compose.dev.yml, update services: test_db to look like the following:
test_db:
image: 'postgres:11'
environment:
POSTGRES_PASSWORD: testpassword
ports:
- '5432:5432'
volumes:
- 'test_db_volume:/var/lib/postgresql/data'
This will provide a password for the postgres database so that it doesn't complain. It looks like this could be easily updated at the source so that when pushkin site default downloads temp.zip, this change is already implemented in the yaml file.
This is a synchronization problem with the CLI. Sometimes, simply running "pushkin init site" again works. Try that and let me know if you get a different error.
What the pushkin init site command does is download all the dependencies the pushkin project needs. You may want to continue the following steps on the quick start and see if things go well.
And for this issue, try to run docker-compose -f pushkin/docker-compose.dev.yml exec -T test_db psql -U postgres -c "create database test_db and docker-compose -f pushkin/docker-compose.dev.yml stop test_db to see if anything change.
Related
I'm sorry for the noob question.
I try to run a PostgreSQL database in a container for tests.
The commands:
docker create --name container-name -p 5432:5432 postgres
docker start container-name
Both commands work but when I check with
docker ps -a
It says, that the container stopped right after the start with exit code 1.
What am I doing wrong?
Thank you very much in advance!!
Thank you #David Maze for your comment.
With docker logs container-name I was able to find out, that I got a way more detailed error message.
It says:
Error: Database is uninitialized and superuser password is not
specified.
You must specify POSTGRES_PASSWORD for the superuser. Use
"-e POSTGRES_PASSWORD=password" to set it in "docker run".
You may also use POSTGRES_HOST_AUTH_METHOD=trust to allow all connections without a password. This is not recommended. See
PostgreSQL
documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html`
This makes it clear for me, that I have to somehow preconfigure the container before I can start it.
I have a docker-compose file that looks like this
version: "3.7"
services:
app:
stdin_open: true
tty: true
build:
context: .
dockerfile: app.Dockerfile
volumes:
- ${HOST_SAVE_DIRC}:${CONTAINER_SAVE_DIRC}
depends_on:
- postgres
postgres:
image: 'postgres'
environment:
- POSTGRES_DB=${POSTGRES_DB}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_HOST_AUTH_METHOD=trust
restart: always
expose:
- "5432"
where variables like POSTGRES_USER are entries from a env file. app.Dockerfile looks like
FROM python:3.8.3-slim-buster
COPY src /src/
COPY init.sql .
COPY .env .
COPY run.sh run.sh
COPY requirements.txt .
RUN ls -a
RUN pip install --no-cache-dir -r requirements.txt
The containers are created, then the user is logged into the app container w/ the main function of the program being called - this is when the database calls
From the app container I am attempting to connect to the postgres container via psycopg2. However when I attempt to do so, I receive the following error:
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host "postgres" (172.22.0.2) and accepting
TCP/IP connections on port 5432?
using a psycopg2 call that looks like
with psy.connect(host='postgres', port=5432, user='postgres', password='postgres') as conn:
...
the entries of this psycopg2 call match the env file given to the docker-compose file.
My understanding is that Postgres uses port 5432 by default. Also that when docker-compose creates the two containers - it creates a docker network for those containers name DIR_default where DIR is the name of the directory the docker-compose file lives in, where each container can be accessed with using the name listed in the docker-compose file ('postgres' and 'app' in these cases).
Among various tries:
I've checked and the database isn't going down between the container being created and the user being exec'd in.
I've tried various little changes like changing the container names, postgres login info, etc.
I've tried linking the postgres container name explicitly with link: "postgres:postgres".
Other solutions suggested here
Any help would be greatly appreciated! I see no reason why something as simple as this should be occurring, but also here I am.
Edit:
Pinging the Postgres container from the app container appears to be working when running docker exec app ping postgres_container_name. Is this a sign that the Docker network is set up correctly and the issue is something of mine?
Edit 2:
Tried clearing all images and containers, then restarting the Docker daemon and afterwards my PC. No change in either case.
For reference, the ping command looked like
docker exec python-app ping name_given_to_postgres_container
returning various statements which looked like
64 bytes from name_given_to_postgres_container.project_name_default (172.18.0.3): icmp_seq=1 ttl=64 time=0.090 ms
which unless I am mistaken, I believe is signalling a succesful ping.
The top level .env file provided to docker-compose
HOST_SAVE_DIRC=~/python_projects/project_directory/directory_in_project
CONTAINER_SAVE_DIRC=/pdfs
POSTGRES_DB=project_name # same as project_directory
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_PORT=5432
Here is the requirements.txt file for the Python app as well
certifi==2020.4.5.1
chardet==3.0.4
idna==2.9
psycopg2-binary==2.8.5
read-env==1.1.0
requests==2.23.0
urllib3==1.25.9
Exec-ing into the Postgres container with docker exec -it container_id bash and running psql -U postgres appears to be successful - even with restart: always removed. I can also see the database named in the docker-compose file is also created. I feel confident in saying this container isn't dying spontaneously.
However, hitting the 5432 port on the Postgres container with netcat via nc name_given_to_postgres_container 5432-5433 returns an error similar to the one returned by psycopg2
arxivist_postgres_1 [172.22.0.3] 5433 (?) : No route to host
arxivist_postgres_1 [172.22.0.3] 5432 (postgresql) : No route to host
The same error is also returned with curl. So my guess the issue isn't with the Postgres container directly, psycopg2, or the host-name - but something with the port?
Edit 3:
As a last attempt to fix this project, the full project this post is referring to is posted at this link. If anyone would like to download the repo and try building the docker containers themselves via ./start.sh - that might be just what is needed to find a solution!
I thought I had Docker setup on my machine, which runs Fedora 32. However as I came to realize from this article, setting up Docker on Fedora 32 requires some extra steps I was not previously aware of.
Specifically for this issue, the command listed in the article to add Docker to whitelist Docker on the local network's firewall with the command
sudo firewall-cmd --permanent --zone=FedoraWorkstation --add-masquerade
So I believe the root cause of my issue was simply my app container being blocked from accessing the postgres container by the firewall. Making the above change made the program work finally!
I have the following docker-compose file:
version: '3'
services:
web:
build:
context: ./django_httpd_mod_wsgi
ports:
- "8000:80"
db:
build:
context: ./postgresql
volumes:
- db-data:/var/lib/postgres/data
volumes:
db-data:
I am building psotgresql image using archlinux:
The following is my postgresql Dockerfile:
FROM archlinux/base
RUN yes | pacman -S postgresql
RUN mkdir /run/postgresql/
RUN chown -R postgres:postgres /run/postgresql/
USER postgres
RUN initdb -D /var/lib/postgres/data
RUN psql -c 'CREATE DATABASE btgapp;'
RUN psql -c "CREATE USER simha WITH PASSWORD 'krishna';"
RUN psql -c 'GRANT ALL PRIVILEGES ON DATABASE btgapp TO simha;'
CMD ["/usr/bin/postgres","-D","/var/lib/postgres/data"]
When i try to do:
docker-compose up
I get the error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/run/postgresql/.s.PGSQL.5432"?
ERROR: Service 'db' failed to build: The command '/bin/sh -c psql -c 'CREATE DATABASE dbname;'' returned a non-zero code: 2
I understood that i have to run the psql -c CREATE DATABSE "dbname" after starting the postgresql server by /usr/bin/postgres -D /var/lib/postgres/data
But i cannot start multiple commands in a Dockerfile. So how to do this.
The option is start a script. But then it will be difficult to see postgres running as a single process.
Based on the comments, I will try to answer here.
I believe that you should go with the postgres 11-alpine image. And I will try to explain why here.
Official docker images come with a number of benefits that you should always consider before starting your own.
Upgrade path is easy - when a new revision of the application wrapped in the image is released, the official docker image will in most cases be updated along with it. And ususally the changes respect the configuration conventions that the image has established. Such as environment variables, startup specifics. So that users can simple change the tag in their stacks, and upgrade. There may of course be breaking changes - always check this.
Large user base - when images like postgres have been downloaded more than 10 milliion times (2019), this does not only mean that it is popular, but inherently works like a guarantee that the image has been tested thoroughly. Any elementary bugs have been weeded out already, and you will have an easy time with the image.
Optimized for size and performance - you can be sure that attention has been paid to a lot of details, minimizing the size of the image and maximizing performance. Many projects publish their applications on a few different linux distros. Like postgres - they publish debian and a alpine based images. The alpine image is the smaller one, while the debian is slightly larger, but gives you access to the vast debian package repositories if you need extra packages installed.
Easy configuration - maintainers of the official images usually understand that usecases of their userbase very well. And they try to make our lives as developers and admins easier (god bless them). Official images usually have some pretty good documentation sitting right on their docker hub landing page, or a link to a github repo where the README.md will cover common usecases. I find that these instructions are worth a good read from top to bottom.
I understand that you want to keep the image small, but what do you know - the postgres project has got your usecase covered.
The latest alpine postgres image tagged 11-alpine has a compressed footprint of 28 MB and decompressed of 70MB. While the archlinux/base image that you want to start off with has compressed base footprint of 153MB and a decompressed size of 445MB. And that's before you introduce postgres itself.
Add to that, that the database and user that you want created on startup - can be handled in the environment variables alone for the official postgres image. Like this:
docker run -d --name some-postgres \
-e POSTGRES_PASSWORD=mysecretpassword \
-e POSTGRES_USER=simha \
-e POSTGRES_DB=btgapp \
postgres:11-alpine
If that does not cover the initialization that you need for your database, then you can copy .sql scripts (and .sh scripts) into a special location in the image - and they will be executed on startup. For this you can extend their image like this:
init-user-db.sh
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE USER simha;
CREATE DATABASE btgapp;
GRANT ALL PRIVILEGES ON DATABASE btgapp TO simha;
EOSQL
And then with a Dockerfile like this:
Dockerfile
FROM postgres:11-alpine
COPY ./init-user-db.sh /docker-entrypoint-initdb.d/init-user-db.sh
(This is taken from the postgres description on docker hub)
In closing - I would recommend that you do not prioritize the distro that an image is based on over the usability and maintainability. Docker enables us to run applications in containers without really caring too much about what distro is inside the container. It's all linux anyway. At the end of the day, I expect that you want a stable postgres database container like me. This is what I get with the official postgres image.
I hope I helped you evaluate your options on this.
I'm a bit new to Docker.
I have two containers running using docker-compose.
One is the API and the other is the actual application.
I want to add a new DB container using the Postgres official image.
It's a bit hard to find a simple tutorial on how to create the container and populate it with a predefined sql file (of schemas and data).
When I start with "CMD /etc/init.d/postgresql start" in the Dockerfile I get an error saying: "No PostgreSQL clusters exist; see "man pg_createcluster" ... (warning)."
Since it takes me too much time to get things going I was wondering if it might be better to get an Ubuntu image and install Postgres on my own since there is only one source on how to use the image - docker hub, and I don't seem to understand it that well.
Any ideas or simple steps on how to compose and 'configure' this image?
If you want populate your database with some file, A simply way to do this is:
How to extend this image
If you would like to do additional initialization in an image derived
from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under
/docker-entrypoint-initdb.d (creating the directory if necessary).
After the entrypoint calls initdb to create the default postgres user
and database, it will run any *.sql files and source any *.sh scripts
found in that directory to do further initialization before starting
the service.
Dockerfile
FROM postgres:alpine
COPY init.sql /docker-entrypoint-initdb.d/init.sql
docker-compose.yml
version: '3'
services:
app:
//your app definition
postgres:
build: .
Pull the postgres image
docker pull postges:14.2
Create the service with the below command
docker service create --name postgres --network my_overlay --env "POSTGRES_PASSWORD=password" --publish 5432:5432 postgres:14.2
Try to connect using userName as postgres and password as password to the default postgres db.
jdbc:postgresql://127.0.0.1:5432/postgres // JDBC connection
I'm looking to build dockerfiles that represent company databases that already exist. Similarly, I'd like create a docker file that starts by restoring a psql dump.
I have my psql_dump.sql in the . directory.
FROM postgres
ADD . /init_data
run "createdb" "--template=template0" "my_database"
run "psql" "-d" "my_database" --command="create role my_admin superuser"
run "psql" "my_database" "<" "init_data/psql_dump.sql"
I thought this would be good enough to do it. I'd like to avoid solutions that use a .sh script. Like this solution.
I use template0 since the psql documentation says you need the same users created that were in the original database, and you need to create the database with template0 before you restore.
However, it gives me an error:
createdb: could not connect to database template1: could not connect to server: No such file or directory
Is the server running locally and accepting
I'm also using docker compose for the overall application, if solving this problem in docker-compose is better, I'd be happy to use the base psql image and use docker compose to do this.
According to the usage guide for the official PostreSQL Docker image, all you need is:
Dockerfile
FROM postgres
ENV POSTGRES_DB my_database
COPY psql_dump.sql /docker-entrypoint-initdb.d/
The POSTGRES_DB environment variable will instruct the container to create a my_database schema on first run.
And any .sql file found in the /docker-entrypoint-initdb.d/ of the container will be executed.
If you want to execute .sh scripts, you can also provide them in the /docker-entrypoint-initdb.d/ directory.
As said in the comments, #Thomasleveil answer is great and simple if your schema recreation is fast.
But in my case it's slow, and I wanted to use docker volumes, so here is what I did
First use docker image as in #Thomasleveil answer to create a container with postgres with all the schema initialization
Dockerfile:
FROM postgres
WORKDIR /docker-entrypoint-initdb.d
ADD psql_dump.sql /docker-entrypoint-initdb.d
EXPOSE 5432
then run it and create new local dir which contains the postgres data after its populated from the “psql_dump.sql” file: docker cp mypg:/var/lib/postgresql/data ./postgres-data
Copy the data to a temp data folder, and start a new postgres docker-compose container whose volume is at the new temp data folder:
startPostgres.sh:
rm -r ./temp-postgres-data/data
mkdir -p ./temp-postgres-data/data
cp -r ./postgres-data/data ./temp-postgres-data/
docker-compose -p mini-postgres-project up
and the docker-compose.yml file is:
version: '3'
services:
postgres:
container_name: mini-postgres
image: postgres:9.5
ports:
- "5432:5432"
volumes:
- ./temp-postgres-data/data:/var/lib/postgresql/data
Now you can run steps #1 and #2 on a new machine or if your psql_dump.sql changes. And each time you want a new clean (but already initialized) db, you can only run startPostgres.sh from step #3.
And it still uses docker volumes.
#Thomasleveil's answer will re-create the database schema at runtime, which is fine for most cases.
If you want to recreate the database schema at buildtime (i.e. if your schema initialization is really slow) you can invoke the stock docker_entrypoint.sh from within your Dockerfile.
However, since the docker_entrypoint.sh is designed to start a long-running database server, you have to add an extra script to exit the process after database initialization but before booting the long-running server.
Dockerfile (with build time database initialization)
# STAGE 1 - Equivalent to #Thomasleveil
FROM postgres AS runtime_init
ENV POSTGRES_DB my_database
COPY 1-psql_dump.sql /docker-entrypoint-initdb.d/
# STAGE 2 - Initialize the database during the build
FROM runtime_init AS buildtime_init_builder
RUN echo "exit 0" > /docker-entrypoint-initdb.d/100-exit_before_boot.sh
ENV PGDATA=/pgdata
RUN docker-entrypoint.sh postgres
# STAGE 3 - Copy the initialized db to a new image to reduce size.
FROM postgres AS buildtime_init
ENV PGDATA=/pgdata
COPY --chown=postgres:postgres --from=buildtime_init_builder /pgdata /pgdata
Important Notes
The stock postgres image will run initialization scripts in alphabetical order, so ensure that your database restoration scripts appear earlier than the exit_before_boot.sh script created in the Dockerfile.
This is demonstrated by the 1 and 100 prefixes shown above. Modify them to your liking.
Database updates to a running instance of this image will not be persisted across reboots since the PGDATA path where the database files are stored no longer maps to a volume mounted from the host machine.
Further Reading
Instructions from the authors of the official postgres image about writing your own custom_entrypoint.sh. This is arguably the more "official" way to solve this problem, but I personally find my approach easier to understand and implement.
A demo of this concept for PostgreSQL 9, which uses the --help flag to exit the docker-entrypoint.sh before the long-running server boots. Unfortunately, this no longer works as of December 3, 2019
Two discussions (1) (2) of this same question from the official docker postgres repository.