I am trying to use services in gitlab ci, namely, postgres. Anyway, postgres service doesn't seem to be running, although I just copied what is there in Gitlab CI docs. In logs, after service logs that it started, some psql command (I don't know where it came from) gives an name resolution error. If I am doing something wrong here, what is the way to run postgres service in gitlab-ci?
Below are .gitlab-ci.yml file and logs:
.gitlab-ci.yml
image: ubuntu
services:
- name: postgres:12.2-alpine
alias: postgres
variables:
POSTGRES_DB: badr
POSTGRES_USER: badr
POSTGRES_PASSWORD: badr
PGHOST: postgres
POSTGRES_HOST_AUTH_METHOD: trust
stages:
- test
test db:
stage: test
before_script:
- until (echo > /dev/tcp/postgres/5432) >/dev/null 2>&1;do >&2 echo "service not ready...sleeping";sleep 5;done
script:
- echo "connected to...$PGHOST"
- sleep 10
logs
Running with gitlab-runner 13.2.0-rc2 (45f2b4ec)
on docker-auto-scale fa6cab46
Preparing the "docker+machine" executor
00:55
Using Docker executor with image ubuntu ...
Starting service postgres:12.2-alpine ...
Pulling docker image postgres:12.2-alpine ...
Using docker image sha256:ae192c4d3adaebbbf2f023e1e50eaadfabccb6b08c855ac13d6ce2232381a58a for postgres:12.2-alpine ...
WARNING: Service postgres:12.2-alpine is already created. Ignoring.
Waiting for services to be up and running...
*** WARNING: Service runner-fa6cab46-project-14794655-concurrent-0-f52b350b86ad38db-postgres-0 probably didn't start properly.
Health check error:
service "runner-fa6cab46-project-14794655-concurrent-0-f52b350b86ad38db-postgres-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2020-07-31T09:15:00.377204181Z ********************************************************************************
2020-07-31T09:15:00.377254629Z WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
2020-07-31T09:15:00.377259167Z anyone with access to the Postgres port to access your database without
2020-07-31T09:15:00.377262471Z a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
2020-07-31T09:15:00.377265670Z documentation about "trust":
2020-07-31T09:15:00.377269181Z https://www.postgresql.org/docs/current/auth-trust.html
2020-07-31T09:15:00.377272282Z In Docker's default configuration, this is effectively any other
2020-07-31T09:15:00.377276152Z container on the same system.
2020-07-31T09:15:00.377295876Z
2020-07-31T09:15:00.377299453Z It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
2020-07-31T09:15:00.377302412Z it with "-e POSTGRES_PASSWORD=password" instead to set a password in
2020-07-31T09:15:00.377305641Z "docker run".
2020-07-31T09:15:00.377308656Z ********************************************************************************
2020-07-31T09:15:00.404620899Z The files belonging to this database system will be owned by user "postgres".
2020-07-31T09:15:00.406021814Z This user must also own the server process.
2020-07-31T09:15:00.406074886Z
2020-07-31T09:15:00.406083517Z The database cluster will be initialized with locale "en_US.utf8".
2020-07-31T09:15:00.406087263Z The default database encoding has accordingly been set to "UTF8".
2020-07-31T09:15:00.406090884Z The default text search configuration will be set to "english".
2020-07-31T09:15:00.406094281Z
2020-07-31T09:15:00.406097490Z Data page checksums are disabled.
2020-07-31T09:15:00.406101511Z
2020-07-31T09:15:00.406197662Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2020-07-31T09:15:00.406858429Z creating subdirectories ... ok
2020-07-31T09:15:00.407274720Z selecting dynamic shared memory implementation ... posix
2020-07-31T09:15:00.428414929Z selecting default max_connections ... 100
2020-07-31T09:15:00.506801199Z selecting default shared_buffers ... 128MB
2020-07-31T09:15:00.689382376Z selecting default time zone ... UTC
2020-07-31T09:15:00.695744690Z creating configuration files ... ok
2020-07-31T09:15:01.009439741Z running bootstrap script ... ok
2020-07-31T09:15:01.355673765Z sh: locale: not found
2020-07-31T09:15:01.355836607Z 2020-07-31 09:15:01.355 UTC [30] WARNING: no usable system locales were found
2020-07-31T09:15:01.784080826Z performing post-bootstrap initialization ... ok
2020-07-31T09:15:02.416545146Z syncing data to disk ... ok
2020-07-31T09:15:02.416652656Z
2020-07-31T09:15:02.416854775Z initdb: warning: enabling "trust" authentication for local connections
2020-07-31T09:15:02.416911707Z You can change this by editing pg_hba.conf or using the option -A, or
2020-07-31T09:15:02.416917642Z --auth-local and --auth-host, the next time you run initdb.
2020-07-31T09:15:02.416962149Z
2020-07-31T09:15:02.416967325Z Success. You can now start the database server using:
2020-07-31T09:15:02.416970415Z
2020-07-31T09:15:02.416990907Z pg_ctl -D /var/lib/postgresql/data -l logfile start
2020-07-31T09:15:02.416995097Z
2020-07-31T09:15:02.440378884Z waiting for server to start....2020-07-31 09:15:02.440 UTC [35] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.2.0) 9.2.0, 64-bit
2020-07-31T09:15:02.442773414Z 2020-07-31 09:15:02.442 UTC [35] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-07-31T09:15:02.461804500Z 2020-07-31 09:15:02.461 UTC [36] LOG: database system was shut down at 2020-07-31 09:15:01 UTC
2020-07-31T09:15:02.465323529Z 2020-07-31 09:15:02.465 UTC [35] LOG: database system is ready to accept connections
2020-07-31T09:15:02.524643142Z done
2020-07-31T09:15:02.524766601Z server started
2020-07-31T09:15:02.537508874Z psql: error: could not connect to server: could not translate host name "postgres" to address: Name does not resolve
*********
Pulling docker image ubuntu ...
Using docker image sha256:1e4467b07108685c38297025797890f0492c4ec509212e2e4b4822d367fe6bc8 for ubuntu ...
Preparing environment
00:02
Getting source from Git repository
00:01
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/badrmoh/cicdtest/.git/
Created fresh repository.
Checking out 604433de as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ until (echo > /dev/tcp/postgres/5432) >/dev/null 2>&1;do >&2 echo "service not ready...sleeping";sleep 5;done
service not ready...sleeping
service not ready...sleeping
service not ready...sleeping
service not ready...sleeping
service not ready...sleeping
service not ready...sleeping
gitlab-runner
The problem was in PGHOST variable. It seems it is used internally by postgres container that is why it fails to start.
The solution is to set PGHOST in script directive:
image: ubuntu
services:
- name: postgres:9
alias: postgres
variables:
POSTGRES_DB: badr
POSTGRES_USER: badr
POSTGRES_PASSWORD: badr
POSTGRES_HOST_AUTH_METHOD: trust
stages:
- test
test db:
stage: test
before_script:
- export PGHOST=postgres
- until (echo > /dev/tcp/$PGHOST/5432) >/dev/null 2>&1;do >&2 echo "service $PGHOST not ready...sleeping";sleep 5;done
script:
- echo "connected to...$PGHOST"
- sleep 10
Note: You can't use variables directive within jobs in this case since it seems to be populated before even starting services itself.
Related
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 months ago.
Improve this question
I am setting up a local Postgres database on Docker with the postgres:14-alpine image, and running database migrations on it with golang-migrate, when I got the following error message after running the migrate tool:
error: pq: role "root" does not exist
I was running the following commands:
$ docker run --name postgres14 -p 5432:5432 -e POSTGRES_USER=root -e POSTGRES_PASSWORD=pass -d postgres:14-alpine
$ docker exec -it postgres14 createdb --user=root --owner=root demodb
$ migrate -path db/migrations -database postgresql://root:pass#localhost:5432/demodb?sslmode=disable --verbose up
These commands can also be viewed in this Makefile, and the full codebase can be found in this repository.
Here are the logs from the Postgres container:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2022-10-15 09:56:41.209 UTC [36] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2022-10-15 09:56:41.211 UTC [36] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-15 09:56:41.217 UTC [37] LOG: database system was shut down at 2022-10-15 09:56:41 UTC
2022-10-15 09:56:41.220 UTC [36] LOG: database system is ready to accept connections
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down...2022-10-15 09:56:41.422 UTC [36] LOG: received fast shutdown request
.2022-10-15 09:56:41.423 UTC [36] LOG: aborting any active transactions
2022-10-15 09:56:41.423 UTC [36] LOG: background worker "logical replication launcher" (PID 43) exited with exit code 1
2022-10-15 09:56:41.424 UTC [38] LOG: shutting down
2022-10-15 09:56:41.434 UTC [36] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
What should I do to configure the root role correctly?
The docker image docs specify that POSTGRES_USER environment variable defaults to postgres if not set, try using that instead of root or drop the container and build it again using the correct environment variable
once you are inside the psql shell you can create a user with
CREATE USER username WITH PASSWORD 'your_password';
then to grant the user access on a specific database:
GRANT ALL PRIVILEGES ON DATABASE demodb TO username;
once that is done you can use the user in the connection string in make file
Turns out the Postgres server that was installed and setup on my OS by Hombrew was using the same port, which clashed with the requests made to the containerized database under the same port number.
This issue can be solved by either using a different port number for the containerized database, or by shutting down the database on the OS.
I am trying to set up a postgres container to start and run initializing the creation of a table. I've succeeded with the straight image from docker but now that I am trying to extend the image a little to create tables when it's produced and I can't get it running. Based off what I've read here How to create User/Database in script for Docker Postgres, this is what I have:
Dockerfile:
FROM library/postgres
COPY init.sql /docker-entrypoint-initdb.d/
init.sql:
CREATE TABLE incident_disposition (
incident_disposition_code VARCHAR,
incident_disposition_code_description VARCHAR
);
From what I understand, FROM library . . . pulls the postgres image from docker hub and the COPY pushes my init.sql script into the entry point so there is no need for a big dockerfile correct?
I then build the image no issue:
Build
docker build -t my_postgres_image .
But when I run I get the issues:
Run
docker run --name testing my_postgres_image --publish 8000:8080 --detach -e POSTGRES_PASSWORD=postgres -d postgres
Errors from logs
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is *not* recommended.
See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
Attempt from comment:
docker container logs testing
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2020-03-26 14:06:51.064 UTC [46] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.072 UTC [46] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.108 UTC [47] LOG: database system was shut down at 2020-03-26 14:06:50 UTC
2020-03-26 14:06:51.119 UTC [46] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE TABLE
2020-03-26 14:06:51.231 UTC [46] LOG: received fast shutdown request
waiting for server to shut down....2020-03-26 14:06:51.232 UTC [46] LOG: aborting any active transactions
2020-03-26 14:06:51.233 UTC [46] LOG: background worker "logical replication launcher" (PID 53) exited with exit code 1
2020-03-26 14:06:51.234 UTC [48] LOG: shutting down
2020-03-26 14:06:51.290 UTC [46] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2020-03-26 14:06:51.345 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-03-26 14:06:51.345 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-03-26 14:06:51.361 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-03-26 14:06:51.387 UTC [64] LOG: database system was shut down at 2020-03-26 14:06:51 UTC
2020-03-26 14:06:51.398 UTC [1] LOG: database system is ready to accept connections
2020-03-26 14:07:27.715 UTC [72] ERROR: relation "incident_disposition" does not exist at character 15
2020-03-26 14:07:27.715 UTC [72] STATEMENT: select * from incident_disposition;
In addition to comments
Due to recent docker image's updates postgres images do not allow to connect to DB without a password from anywhere. So you need to specify username/password
docker run -p 8000:8080 -e POSTGRES_PASSWORD=postgres --name testing -d my_postgres_image
Or if you still don't want to use password, you can just set POSTGRES_HOST_AUTH_METHOD=trust environment variable:
docker run -p 8000:8080 -e POSTGRES_HOST_AUTH_METHOD=trust --name testing -d my_postgres_image
It is a typical Initialization scripts issue.
You can file the full explaination in postgresql docker page. https://hub.docker.com/_/postgres
Here is the brief intro:
1. One common problem is that if one of your /docker-entrypoint-initdb.d scripts fails (which will cause the entrypoint script to exit) and your orchestrator restarts the container with the already initialized data directory, it will not continue on with your scripts.
note:
in your case, you may need clean the historical docker containers(stopped) by
step 1: docker ps |grep
step 2: docker rm -f -v
Or if you are using docker-compose, the historical orchestrator could be easily removed by docker-compose down -v.
I have a docker compose file where i am launching PostgreSQL with shared volume. But i am continuously getting the bellow ERROR.
2018-10-11 14:57:01.757 GMT [81] LOG: skipping missing configuration
file "/postgresql/data/postgresql.auto.conf"
| 2018-10-11 14:57:01.768 GMT [81] FATAL: data directory "/postgresql/data" has wrong ownership
| 2018-10-11 14:57:01.768 GMT [81] HINT: The server must be started by the user that owns the data directory.
My docker compose file as below
addb:
image : postgres
networks:
- private
ports:
- "5432:5432"
volumes:
- /mnt/shared/app_data/db/postgres/data_db:/postgresql/data
depends_on:
- sol-server
I am using RHEL 7.5 and Docker version 18.06.1-ce, build e68fc7. Any Idea how i can resolve the above issue.
You should mount your volume with the database in /var/lib/postgresql/data
instead of /postgresql/data or give a enviroment PGDATA=/postgresql/data
I was having issues connecting to my PG database in my compose file. I finally eliminated all other factors and made this bare-bones compose file:
version: '2.1'
services:
database:
image: postgres:9.5.6
environment:
POSTGRES_PASSWORD: secretpass
ports:
- 5432:5432
I then started it up with this command:
docker-compose --file docker-compose.pg.yml up --build
and it seemed to start ok:
Starting app_database_1 ...
Starting app_database_1 ... done
Attaching to app_database_1
database_1 | LOG: database system was shut down at 2017-09-22 18:48:55 UTC
database_1 | LOG: MultiXact member wraparound protections are now enabled
database_1 | LOG: database system is ready to accept connections
database_1 | LOG: autovacuum launcher started
Then from another shell, tried to connect with this command:
psql postgresql://postgres:secretpass#0.0.0.0
And got this output on my first shell where the compose was run:
database_1 | FATAL: password authentication failed for user "postgres"
database_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all 0.0.0.0/0 md5"
I have also tried to connect from another container running
Rails defined in the compose file and get the same error. I always end up with an authentication error. What am I missing?
When using the https://github.com/kiasaki/docker-alpine-postgres image in docker compose, the scripts inside /docker-entrypoint.initdb.d are not being executed as I view from the logs:
db_1 | LOG: database system was shut down at 2016-09-05 18:16:02 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
The thing is, if I build the Dockerfile standalone and run the container, it will execute the sql files inside the folder, like you can see from the logs:
waiting for server to start....LOG: database system was shut down at 2016-09-05 18:28:18 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/syna-setup.sql
CREATE TABLE
INSERT 0 1
CREATE TABLE
waiting for server to shut down...LOG: received fast shutdown request
.LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
done
server stopped
LOG: database system was shut down at 2016-09-05 18:28:21 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
The docker-compose.yml:
version: '2'
services:
db:
build: ./db
environment:
- POSTGRES_PASSWORD=testpass
restart: always
app:
build: ./app
ports:
- "5000:5000"
volumes:
- .:/app/src
depends_on:
- db
environment:
- DB_SERVER=postgres://postgres:testpass#db:5432/postgres
What am I doing wrong here?
UPDATED as he use his own Dockerfile, not from the github link.
Could you proide the docker run command so that if I build the Dockerfile standalone and run the container, it will execute the sql files inside the folder. You should mount some directory to the container so it can find the SQL files.
I read the Dockerfile at https://github.com/kiasaki/docker-alpine-postgres/blob/master/Dockerfile and asked myself where are the SQL files? It just make the directory without copy SQL files
mkdir /docker-entrypoint-initdb.d
Then, in the docker-entrypoint.sh, it check if there's any SQL file to execute
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh) echo "$0: running $f"; . "$f" ;;
*.sql) echo "$0: running $f"; psql --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" < "$f" && echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done
I think we should mount a SQL directory, like
version: '2'
services:
db:
build: ./db
environment:
- POSTGRES_PASSWORD=testpass
restart: always
entrypoint:
- docker-entrypoint.sh
volumes:
- ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
- ./docker-entrypoint.sh:/docker-entrypoint.sh