Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
So, I need docker image with: my program that use postgresql, postgresql and some data in it
I use this article: https://medium.com/#sharmaNK/build-postgres-docker-image-with-data-included-489bd58a1f9e
My steps:
docker-compose build //Creates ps_image_with_data
docker run --name ps-export -p 5432:5432 -i postgres
Connect to 5432 db, using dbeaver; create table; put some rows in the table
docker exec -it my_container_with_data_id bash
mkdir /postgres
cp -r /var/lib/postgresql/data/* /postgres
docker commit ps-export ps_image_with_data
docker run --env PGDATA=postgres -p 5432:5432 -i ps_image_with_data // here I've tried both ps_image_with_data and new image that was created on step 7
And after 8 step I got this:
2018-11-08 12:44:52.549 UTC [1] LOG: listening on IPv4 address
"0.0.0.0", port 5432 2018-11-08 12:44:52.549 UTC [1] LOG: listening
on IPv6 address "::", port 5432 2018-11-08 12:44:52.615 UTC [1] LOG:
listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2018-11-08 12:44:52.750 UTC [22] LOG: database system was
interrupted; last known up at 2018-11-08 11:56:27 UTC 2018-11-08
12:45:25.042 UTC [22] LOG: could not remove cache file
"global/pg_internal.init": Permission denied 2018-11-08 12:45:25.042
UTC [22] LOG: could not remove cache file
"base/13067/pg_internal.init": Permission denied 2018-11-08
12:45:25.042 UTC [22] LOG: database system was not properly shut
down; automatic recovery in progress 2018-11-08 12:45:25.127 UTC [22]
LOG: redo starts at 0/166CF68 2018-11-08 12:45:25.127 UTC [22] LOG:
invalid record length at 0/166D048: wanted 24, got 0 2018-11-08
12:45:25.127 UTC [22] LOG: redo done at 0/166D010 2018-11-08
12:45:25.259 UTC [22] PANIC: could not rename file
"pg_logical/replorigin_checkpoint.tmp" to
"pg_logical/replorigin_checkpoint": Permission denied 2018-11-08
12:45:25.357 UTC [1] LOG: startup process (PID 22) was terminated by
signal 6: Aborted 2018-11-08 12:45:25.358 UTC [1] LOG: aborting
startup due to startup process failure 2018-11-08 12:45:25.359 UTC [1]
LOG: database system is shut down
Can someone help me to sort it out?
How about
docker run --name ps-export -p 5432:5432 -i postgres
Connect to 5432 db, using dbeaver; create table; put some rows in the table
Make a dump (data.sql) to your host
Add in your docker-compose.yml, under the psql server
`
volumes:
- ./data.sql:/docker-entrypoint-initdb.d/init.sql
docker-compose up
If you want to include data.sql in the image, instead of step 4, copy it to /docker-entrypoint-initdb.d/init.sql in your dockerfile.
Related
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 months ago.
Improve this question
I am setting up a local Postgres database on Docker with the postgres:14-alpine image, and running database migrations on it with golang-migrate, when I got the following error message after running the migrate tool:
error: pq: role "root" does not exist
I was running the following commands:
$ docker run --name postgres14 -p 5432:5432 -e POSTGRES_USER=root -e POSTGRES_PASSWORD=pass -d postgres:14-alpine
$ docker exec -it postgres14 createdb --user=root --owner=root demodb
$ migrate -path db/migrations -database postgresql://root:pass#localhost:5432/demodb?sslmode=disable --verbose up
These commands can also be viewed in this Makefile, and the full codebase can be found in this repository.
Here are the logs from the Postgres container:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2022-10-15 09:56:41.209 UTC [36] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2022-10-15 09:56:41.211 UTC [36] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-15 09:56:41.217 UTC [37] LOG: database system was shut down at 2022-10-15 09:56:41 UTC
2022-10-15 09:56:41.220 UTC [36] LOG: database system is ready to accept connections
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down...2022-10-15 09:56:41.422 UTC [36] LOG: received fast shutdown request
.2022-10-15 09:56:41.423 UTC [36] LOG: aborting any active transactions
2022-10-15 09:56:41.423 UTC [36] LOG: background worker "logical replication launcher" (PID 43) exited with exit code 1
2022-10-15 09:56:41.424 UTC [38] LOG: shutting down
2022-10-15 09:56:41.434 UTC [36] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
What should I do to configure the root role correctly?
The docker image docs specify that POSTGRES_USER environment variable defaults to postgres if not set, try using that instead of root or drop the container and build it again using the correct environment variable
once you are inside the psql shell you can create a user with
CREATE USER username WITH PASSWORD 'your_password';
then to grant the user access on a specific database:
GRANT ALL PRIVILEGES ON DATABASE demodb TO username;
once that is done you can use the user in the connection string in make file
Turns out the Postgres server that was installed and setup on my OS by Hombrew was using the same port, which clashed with the requests made to the containerized database under the same port number.
This issue can be solved by either using a different port number for the containerized database, or by shutting down the database on the OS.
I have a docker instance and I have changed the file =:
"/var/lib/postgreqsql/data/pg_hba.conf"
And it now has errors.
It appears that this is preventing me actually connecting to the docker instance.
docker logs --tail 20 postgres
2022-08-04 00:30:26.392 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-04 00:30:26.392 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-04 00:30:26.417 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-04 00:30:26.451 UTC [1] LOG: end-of-line before authentication method
2022-08-04 00:30:26.451 UTC [1] CONTEXT: line 87 of configuration file "/var/lib/postgreqsql/data/pg_hba.conf"
2022-08-04 00:30:26.451 UTC [1] FATAL: could not load pg_hba.conf
2022-08-04 00:30:26.453 UTC [1] LOG: database system is shut down
When I try and connect I get a consistent message as follows:
root#VM01:~/vm01-docker# docker exec -u root -it postgres bash
Error response from daemon: Container 4545dcba980215c is restarting, wait until the container is running
root#VM01:~/vm01-docker#
Stopping and starting as follows:
docker-compose stop postgres
docker-compose start postgres
Has no impact the issue remains.
What I need to do is be able to connect again to the instance and edit the file pg_hba.conf again to correct the fault but I am not sure how I can. (I am a docker newb.)
I am trying to set up a postgres container to start and run initializing the creation of a table. I've succeeded with the straight image from docker but now that I am trying to extend the image a little to create tables when it's produced and I can't get it running. Based off what I've read here How to create User/Database in script for Docker Postgres, this is what I have:
Dockerfile:
FROM library/postgres
COPY init.sql /docker-entrypoint-initdb.d/
init.sql:
CREATE TABLE incident_disposition (
incident_disposition_code VARCHAR,
incident_disposition_code_description VARCHAR
);
From what I understand, FROM library . . . pulls the postgres image from docker hub and the COPY pushes my init.sql script into the entry point so there is no need for a big dockerfile correct?
I then build the image no issue:
Build
docker build -t my_postgres_image .
But when I run I get the issues:
Run
docker run --name testing my_postgres_image --publish 8000:8080 --detach -e POSTGRES_PASSWORD=postgres -d postgres
Errors from logs
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is *not* recommended.
See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
Attempt from comment:
docker container logs testing
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2020-03-26 14:06:51.064 UTC [46] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.072 UTC [46] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.108 UTC [47] LOG: database system was shut down at 2020-03-26 14:06:50 UTC
2020-03-26 14:06:51.119 UTC [46] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE TABLE
2020-03-26 14:06:51.231 UTC [46] LOG: received fast shutdown request
waiting for server to shut down....2020-03-26 14:06:51.232 UTC [46] LOG: aborting any active transactions
2020-03-26 14:06:51.233 UTC [46] LOG: background worker "logical replication launcher" (PID 53) exited with exit code 1
2020-03-26 14:06:51.234 UTC [48] LOG: shutting down
2020-03-26 14:06:51.290 UTC [46] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2020-03-26 14:06:51.345 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-03-26 14:06:51.361 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.387 UTC [64] LOG: database system was shut down at 2020-03-26 14:06:51 UTC
2020-03-26 14:06:51.398 UTC [1] LOG: database system is ready to accept connections
2020-03-26 14:07:27.715 UTC [72] ERROR: relation "incident_disposition" does not exist at character 15
2020-03-26 14:07:27.715 UTC [72] STATEMENT: select * from incident_disposition;
In addition to comments
Due to recent docker image's updates postgres images do not allow to connect to DB without a password from anywhere. So you need to specify username/password
docker run -p 8000:8080 -e POSTGRES_PASSWORD=postgres --name testing -d my_postgres_image
Or if you still don't want to use password, you can just set POSTGRES_HOST_AUTH_METHOD=trust environment variable:
docker run -p 8000:8080 -e POSTGRES_HOST_AUTH_METHOD=trust --name testing -d my_postgres_image
It is a typical Initialization scripts issue.
You can file the full explaination in postgresql docker page. https://hub.docker.com/_/postgres
Here is the brief intro:
1. One common problem is that if one of your /docker-entrypoint-initdb.d scripts fails (which will cause the entrypoint script to exit) and your orchestrator restarts the container with the already initialized data directory, it will not continue on with your scripts.
note:
in your case, you may need clean the historical docker containers(stopped) by
step 1: docker ps |grep
step 2: docker rm -f -v
Or if you are using docker-compose, the historical orchestrator could be easily removed by docker-compose down -v.
I'd like to try out the quick start example at https://github.com/wrouesnel/postgres_exporter on a MacOS host device. In one terminal, I run a postgres image in interactive mode on the host network:
> docker run --network=host -it --rm --env POSTGRES_PASSWORD=password postgres
Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
8f91359f1fff: Pull complete
c6115f5efcde: Pull complete
28a9c19d8188: Pull complete
2da4beb7be31: Pull complete
fb9ca792da89: Pull complete
cedc20991511: Pull complete
b866c2f2559e: Pull complete
5d459cf6645c: Pull complete
cf66247ad881: Pull complete
35e29440d9da: Pull complete
362779251360: Pull complete
fb82d778d08d: Pull complete
b1f8d21ff25d: Pull complete
6d49eb0e8dd0: Pull complete
Digest: sha256:be456a40361cd836e0e1b35fc4d872e20e138f214c93138425169c4a2dfe1b0e
Status: Downloaded newer image for postgres:latest
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2019-10-03 19:06:10.222 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-03 19:06:10.235 UTC [43] LOG: database system was shut down at 2019-10-03 19:06:10 UTC
2019-10-03 19:06:10.239 UTC [42] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down....2019-10-03 19:06:10.320 UTC [42] LOG: received fast shutdown request
2019-10-03 19:06:10.322 UTC [42] LOG: aborting any active transactions
2019-10-03 19:06:10.325 UTC [42] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
2019-10-03 19:06:10.325 UTC [44] LOG: shutting down
2019-10-03 19:06:10.338 UTC [42] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2019-10-03 19:06:10.436 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2019-10-03 19:06:10.438 UTC [1] LOG: listening on IPv6 address "::", port 5432
2019-10-03 19:06:10.440 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-03 19:06:10.450 UTC [51] LOG: database system was shut down at 2019-10-03 19:06:10 UTC
2019-10-03 19:06:10.453 UTC [1] LOG: database system is ready to accept connections
while in another terminal, I run postgres_exporter and connect it to that database:
> docker run --net=host --env DATA_SOURCE_NAME="postgresql://postgres:password#localhost:5432/postgres?sslmode=disable" wrouesnel/postgres_exporter
Unable to find image 'wrouesnel/postgres_exporter:latest' locally
latest: Pulling from wrouesnel/postgres_exporter
0d6d2d2516f9: Pull complete
e9d7b571ef5e: Pull complete
Digest: sha256:aeea975f0efeacb49c170f0f7c4a4000d3f0099cc33437aedd3f276e628cde1c
Status: Downloaded newer image for wrouesnel/postgres_exporter:latest
time="2019-10-03T19:09:20Z" level=info msg="Established new database connection to \"localhost:5432\"." source="postgres_exporter.go:778"
time="2019-10-03T19:09:20Z" level=info msg="Semantic Version Changed on \"localhost:5432\": 0.0.0 -> 11.5.0" source="postgres_exporter.go:1238"
time="2019-10-03T19:09:20Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1459"
As I understand it, I should be able to go to localhost:9187 in my browser and see the exported metrics. What I get, however, is that the connection gets refused:
> curl http://localhost:9187
curl: (7) Failed to connect to localhost port 9187: Connection refused
What I suspect is that this is because, as documented at https://docs.docker.com/network/host/,
The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
Is this what is causing the issue and if so, how can I adapt this quickstart example to work using Docker Desktop for Mac?
A little late, but maybe for someone in the future. I had the same problem on Docker Desktop for Windows. For it to work I used the following command:
docker run -p 9187:9187 -e DATA_SOURCE_NAME="postgresql://postgres:password#localhost:5432/postgres?sslmode=disable" wrouesnel/postgres_exporter
We built a debian docker image with postgresql to run one of our service. The database is for internal container use and does not need port mapping. I believe it is installed via apt-get in the Dockerbuild file.
We stop and start this service often, and it is a performance issue that the database is slow to startup. Although empty, takes sightly over 20s to accept connection on the first time we start the docker image. The log is as follow :
2019-04-05 13:05:30.924 UTC [19] LOG: could not bind IPv6 socket: Cannot assign requested address
2019-04-05 13:05:30.924 UTC [19] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-04-05 13:05:30.982 UTC [20] LOG: database system was shut down at 2019-04-05 12:57:16 UTC
2019-04-05 13:05:30.992 UTC [20] LOG: MultiXact member wraparound protections are now enabled
2019-04-05 13:05:30.998 UTC [19] LOG: database system is ready to accept connections
2019-04-05 13:05:30.998 UTC [24] LOG: autovacuum launcher started
2019-04-05 13:05:31.394 UTC [26] [unknown]#[unknown] LOG: incomplete startup packet
2019-04-19 13:21:58.974 UTC [37] LOG: could not bind IPv6 socket: Cannot assign requested address
2019-04-19 13:21:58.974 UTC [37] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-04-19 13:21:59.025 UTC [38] LOG: database system was interrupted; last known up at 2019-04-05 13:05:34 UTC
2019-04-19 13:21:59.455 UTC [39] [unknown]#[unknown] LOG: incomplete startup packet
2019-04-19 13:21:59.971 UTC [42] postgres#postgres FATAL: the database system is starting up
[...]
2019-04-19 13:22:15.221 UTC [85] root#postgres FATAL: the database system is starting up
2019-04-19 13:22:15.629 UTC [38] LOG: database system was not properly shut down; automatic recovery in progress
2019-04-19 13:22:15.642 UTC [38] LOG: redo starts at 0/14EEBA8
2019-04-19 13:22:15.822 UTC [38] LOG: invalid record length at 0/24462D0: wanted 24, got 0
2019-04-19 13:22:15.822 UTC [38] LOG: redo done at 0/24462A8
2019-04-19 13:22:15.822 UTC [38] LOG: last completed transaction was at log time 2019-04-05 13:05:36.602318+00
2019-04-19 13:22:16.084 UTC [38] LOG: MultiXact member wraparound protections are now enabled
2019-04-19 13:22:16.094 UTC [37] LOG: database system is ready to accept connections
2019-04-19 13:22:16.094 UTC [89] LOG: autovacuum launcher started
2019-04-19 13:22:21.528 UTC [92] root#test LOG: could not receive data from client: Connection reset by peer
2019-04-19 13:22:21.528 UTC [92] root#test LOG: unexpected EOF on client connection with an open transaction
Any suggetion in fixing this startup issue ?
EDIT : Some requested the dockerfile, here is relevant lines
RUN apt-get update \
&& apt-get install -y --force-yes \
postgresql-9.6-pgrouting \
postgresql-9.6-postgis-2.3 \
postgresql-9.6-postgis-2.3-scripts \
[...]
# Download, compile and install GRASS 7.2
[...]
USER postgres
# Create a database 'grass_backend' owned by the "root" role.
RUN /etc/init.d/postgresql start \
&& psql --command "CREATE USER root WITH SUPERUSER [...];" \
&& psql --command "CREATE EXTENSION postgis; CREATE EXTENSION plpython3u;" --dbname [dbname] \
&& psql --command "CREATE EXTENSION postgis_sfcgal;" --dbname [dbname] \
&& psql --command "CREATE EXTENSION postgis; CREATE EXTENSION plpython3u;" --dbname grass_backend
WORKDIR [...]
End of file after workdir, meaning I guess the database isn't properly shut down
Answer I stopped properly postgresql inside the docker install. It now starts 15s faster. Thanks for replying
Considering the line database system was not properly shut down; automatic recovery in progress that would definitely explain slow startup, please don't kill the service, send the stop command and wait for it to close properly.
Please note that the system might kill the process if it takes to long to stop, this will happen in the case of postgresql if there are connections still held to it (probably from your application). If you disconnect all the connections and than stop, postgresql should be able to stop relatively quickly.
Also make sure you stop the postgresql service inside the container before turning it off.
TCP will linger connections for a while, if you are starting and stopping in quick succession without properly stopping the service inside that would explain your error of why the port is unavailable, normally the service can start/stop in very quick succession on my machine if nothing is connected to it.
3 start-stop cycles of postgresql on my machine (I have 2 decently sized databases)
$ time bash -c 'for i in 1 2 3; do /etc/init.d/postgresql-11 restart; done'
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
real 0m1.188s
user 0m0.260s
sys 0m0.080s