I am trying to setup GitLab using separate docker containers for both GitLab and PostgreSQL. I am using RancherOS v1.0.3 with Kubernetes enabled. For now, all i want is to have a single node having both the containers. Later, i will look into configuring them on separate nodes.
[rancher#rancher-agent-2 ~]$ cat postgresql.sh
docker run --name=postgresql -d \
--env 'DB_NAME=gitlabhq_production' \
--env 'DB_USER=gitlab' --env 'DB_PASS=password' \
--env 'DB_EXTENSION=pg_trgm' \
--volume /opt/postgresql:/var/lib/postgresql \
postgres:9.6.3
[rancher#rancher-agent-2 ~]$ cat gitlab.sh
docker run --name=gitlab -d --link postgresql:postgresql \
-v /opt/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:9.3.9-ce.0
Now when i run the PostgreSQL container and try logging in, i get errors.
[rancher#rancher-agent-2 ~]$ docker exec -it postgresql bash
root#a6cef780c594:/# psql -d gitlabhq_production -U gitlab
psql: FATAL: role "gitlab" does not exist
Seems the db (gitlabhq_production) has not been created.
root#a6cef780c594:/# psql -U postgres
psql (9.6.3)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Why is it not creating the db despite passing the parameters?
UPDATE:
PostgreSQL container log:
[rancher#rancher-agent-2 ~]$ docker logs postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
...
...
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
ALTER ROLE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
...
...
You are using the wrong environment vars names:
DB_NAME should be POSTGRES_DB
DB_USER should be POSTGRES_USER
DB_PASS should be POSTGRES_PASSWORD
Check https://hub.docker.com/_/postgres/
Related
I updated my docker file to version 3 but now the inicial schemas are not being created. i already tried with a different volume
I run it with : docker-compose -f docker-compose.yml up
version: '3'
services:
db-service:
image: postgres:11.2
volumes:
- ./dbscripts/:/docker-entrypoint-initdb.d/
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=sample
ports:
- 5432:5432
Check these 2 things:
The init scripts are triggered only on the first deploy. On the subsequent docker-compose ups you probably see as well on the logs the following message:
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
To force the re-initialization, you can wipe the data volume(This will delete the entries, not only the schema).
To find the path, simply docker inspect <postgres-container-id> and look for the HostConfig.Binds
The script executes till the first type "longvarbinary" does not exist error. You will need to fix this.
db-service_1 | CREATE TABLE
db-service_1 | 2020-07-26 19:29:34.121 UTC [68] ERROR: type "longvarbinary" does not exist at character 68
I opened a console in the postgres container and your sample database is there, as well as a table that is created before the .sql error occurs in the script:
docker exec -ti 22b0bb87561f sh
# psql -U postgres
psql (11.2 (Debian 11.2-1.pgdg90+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
sample | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# \c sample
You are now connected to database "sample" as user "postgres".
sample=# \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------+-------+----------
public | oauth_client_details | table | postgres
(1 row)
I ran into the same issue, and I figured it was because I forgot to clear the volume when I take down docker compose with command docker-compose down.
The solution that helps is when you take down docker compose, you specify that all volumes must be removed docker-compose down --volumes. Also, make sure to check and delete any related volumes using docker volume ls.
WARNING: Any subsequent command docker-compose down should not have the flag --volume or else your persistent data will be wiped out. This may or may not be desirable depending on your use case. The idea is to ONLY run the init script on the very first time you run docker-compose up. Any subsequent command up after that will skip initialization because data is already populated, and you don't want to override it.
Hope it helps!
I want to build a custom Postgres11 image in which some users are created and some extensions are installed. As I want these to be created at build time, so I don't want to use docker-entrypoint-initdb.d. A next step would be to restore a sql dump as well.
FROM postgres:11
ENV PG_MAJOR 11
ENV POSTGISV 2.5
ENV TZ Europe/Brussels
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGISV \
postgresql-$PG_MAJOR-postgis-$POSTGISV-scripts
USER postgres
RUN initdb && pg_ctl -o "-c listen_addresses='*'" start &&\
psql -h 0.0.0.0 --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
psql -h 0.0.0.0 --command "CREATE USER akela_test WITH PASSWORD 'akela';" &&\
createdb -E UTF8 -U postgres -h 0.0.0.0 -O akela_test akela_test --template template0 &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "hstore";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "postgis";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "CREATE ROLE akela_db WITH LOGIN PASSWORD 'akela'" &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "GRANT ALL PRIVILEGES ON DATABASE akela_test to akela_db" &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "CREATE schema db" &&\
pg_ctl stop
# gunzip -c /tmp/dump.sql.gz | psql -U akela -h 0.0.0.0 akela
USER root
seems to work:
...
CREATE SCHEMA
ALTER SCHEMA
CREATE ROLE
GRANT
CREATE SCHEMA
ALTER SCHEMA
waiting for server to shut down....2019-07-08 12:58:06.962 CEST [22] LOG: received fast shutdown request
2019-07-08 12:58:06.964 CEST [22] LOG: aborting any active transactions
2019-07-08 12:58:06.965 CEST [22] LOG: background worker "logical replication launcher" (PID 29) exited with exit code 1
2019-07-08 12:58:06.965 CEST [24] LOG: shutting down
2019-07-08 12:58:07.006 CEST [22] LOG: database system is shut down
done
server stopped
...
running the image however shows no users nor db:
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
postgres=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
What could be the issue?
The Dockerfile for postgres defines a volume which means any changes to this directory by a RUN step will be discarded. To make changes to this directory you need to do one of the following options:
Make the changes at run time rather than doing the build, and save the resulting volume.
Make changes during the build, but in a different directory. This would require changing the postgres configuration to use the different directory.
Save your changes to a different directory and then restore those changes when you start the container (see the save and load volume scripts for an example of this).
Build your own postgres image without the volume definition.
I have wasted an entire day on this, and to say I'm not impressed by the unnecessary complexity of what should be a simple task - would be a gross understatement.
Ok, having got that off my chest, I am building a django application using docker-machine, docker-compose, postgresql and redis - by following this tutorial.
I have managed to get the basic tutorial to work - but it does not suit my needs, as I need to create a user and a database for my application - as opposed to using 'postgres' for both.
I have used the answer from #dnephin to a similar question, and modified my code as follows:
I created a new Dockerfile in a new directory ./database/:
FROM postgres:9.6
COPY . /fixtures
WORKDIR /fixtures
RUN /fixtures/setup.sh
./database/setup.sh contents:
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
/etc/init.d/postgresql start
su - postgres # makes no effing difference ...
psql -f create_fixtures.sql
/etc/init.d/postgresql stop
./database/create_fixtures.sql contents:
CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
CREATE USER webuser ENCRYPTED PASSWORD 'deadbeefsnaf0' NOSUPERUSER NOCREATEDB NOCREATEROLE;
GRANT ALL PRIVILEGES ON mydatabase TO webuser;
and finally my postgres service in the docker_compose.yml is modified to use build:
postgres:
build: ./database/
...
When I run docker-compose build, the build goes through the motions and then barfs at where I'm importing the SQL fixtures file via psql:
frothing#themouth:~/path/to/directory$ docker-compose build
redis uses an image, skipping
Building postgres
Step 1/4 : FROM postgres:9.6
---> ff0943ecbb3c
Step 2/4 : COPY . /fixtures
---> fae19dc88da8
Removing intermediate container 84b860aee55c
Step 3/4 : WORKDIR /fixtures
---> aa88438dc69f
Removing intermediate container b801ddc3b374
Step 4/4 : RUN /fixtures/setup.sh
---> Running in ca3e89ec2460
Creating new cluster 9.6/main ...
config /etc/postgresql/9.6/main
data /var/lib/postgresql/9.6/main
locale en_US.utf8
socket /var/run/postgresql
port 5432
Starting PostgreSQL 9.6 database server: main.
psql: FATAL: role "root" does not exist
ERROR: Service 'postgres' failed to build: The command '/bin/sh -c /fixtures/setup.sh' returned a non-zero code: 2
I tried to solve this using the useless documentation on docker for postgresql service - but got no where.
How can I solve this?
Volumes are not available at build time. You can create /var/lib/postgresql/data in your script but it will be overwritten by VOLUME /var/lib/postgresql/data from postgres:9.6 image.
In your case: just use the following docker file:
FROM postgres:9.6
COPY ./create_fixtures.sql /docker-entrypoint-initdb.d/create_fixtures.sql
They get automatically executed once the container starts. Here is an example:
$ docker run -d --name mydb -p 33306:3306 yourtag
$ docker exec -ti mydb psql -U postgres
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------+----------+----------+------------+------------+-----------------------
mydatabase | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | webuser=CTc/postgres
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Outdated answer:
Your script should work on a container except in the fixture you have to execute psql like this:
su postgres -c "psql -f create_fixtures.sql"
su --login postgres does not work because postgres can't open a bash or shell. You can try around with docker run --rm -ti postgres:9.6 bash.
Sorry I have to tell you there is one more error in your sql script: GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser - the keyword DATABASE is necessary here.
Here is a complete log how I tested and can confirm this works:
docker run --rm -ti postgres:9.6 bash
root#be03ab1eb704:/# cat > test.sql <<EOF
> CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
> CREATE USER webuser ENCRYPTED PASSWORD 'asdf123' NOSUPERUSER NOCREATEDB NOCREATEROLE;
> GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser;
> EOF
root#be03ab1eb704:/# pg_createcluster 9.6 main --start
Creating new PostgreSQL cluster 9.6/main ...
/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/9.6/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/9.6/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/9.6/bin/pg_ctl -D /var/lib/postgresql/9.6/main -l logfile start
Ver Cluster Port Status Owner Data directory Log file
9.6 main 5432 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
root#be03ab1eb704:/# /etc/init.d/postgresql start
[ ok ] Starting PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# su postgres -c "psql -f test.sql"
CREATE DATABASE
CREATE ROLE
GRANT
root#be03ab1eb704:/# /etc/init.d/postgresql stop
[ ok ] Stopping PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# exit
exit
The official postgresql docker image automatically imports scripts at the first start of a container. So if you mount your directory with your init sql script to container path '/docker-entrypoint-initdb.d/' it should be run.
For example if you have your import script myImport.sql and it's on your host in a directory /opt/import, you can mount the import directory on container start to your postgres image to /docker-entrypoint-initdb.d and the sql file will be executed after initial database setup.
docker run -p $toHostParam:5432 -e POSTGRES_PASSWORD="$dbPwd" \
-e POSTGRES_USER="$dbUser" \
-e POSTGRES_DB="$dbName" \
-e PGDATA=/opt/pgdata \
-v ${dataDir}:/opt/pgdata \
# look here
# mount of local import dir
-v /opt/import:/docker-entrypoint-initdb.d \
postgres:9.6
Take a look at the postgesql image start script here (from line 126):
https://github.com/docker-library/postgres/blob/master/9.6/docker-entrypoint.sh
If you want a specific db user or database you can also customize your postgresql container with environment variables.
Take a look at the 'Environment Variables' section here: https://hub.docker.com/_/postgres/
Try with this setup.sh
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
su postgres sh -c "/etc/init.d/postgresql start && psql -f create_fixtures.sql && /etc/init.d/postgresql stop"
Try a explicit user when running psql:
psql -U postgres -f create_fixtures.sql
I'm trying to set up two databases on travis but it just stops halfway the before_install stating:
(END)
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
eg https://travis-ci.org/B3Partners/brmo/builds/85746119
my yml is the following:
language: java
sudo: false
branches:
only:
- travis-integration
addons:
postgresql: "9.3"
jdk:
# - openjdk6
# - openjdk7
- oraclejdk7
# - oraclejdk8
matrix:
fast_finish: true
cache:
directories:
- $HOME/.m2
before_install:
# STAGING
- psql --version
- psql -U postgres -c 'create database staging'
- psql -U postgres -c 'create database rsgb'
- psql -U postgres --list
# set up RSGB
- psql -U postgres -d rsgb -c 'create extension postgis'
- psql -U postgres -d rsgb -f ./datamodel/generated_scripts/datamodel_postgresql.sql --single-transaction --echo-all
# - psql -f ./datamodel/utility_scripts/111a_update_gemeente_geom.sql -U postgres -d rsgb --single-transaction
# - psql -f ./datamodel/utility_scripts/113a_update_wijk_geom.sql -U postgres -d rsgb --single-transaction
install:
# install all dependencies + artifacts without any testing
- mvn install -Dmaven.test.skip=true -B -V -fae -q
before_script:
# dit dient na afloop van de 'install' gedaan te worden omdat de staging DB sql gegenereerd wordt door Hibernate
- psql -U postgres -d staging -f ./brmo-persistence/target/ddlscripts/create-brmo-persistence-postgresql.sql --single-transaction
- psql -U postgres -d staging -f ./brmo-persistence/db/01_create_indexes.sql
- psql -U postgres -d staging -f ./brmo-persistence/db/02_insert_default_user.sql
- psql -U postgres -d staging -f ./brmo-persistence/db/03_update_status_enum_value.sql
# run tests
script:
# run unit tests
- mvn -e test -B
# run integration tests
- mvn -e verify -B
after_success:
after_failure:
after_script:
notifications:
email: false
# on_success: [always|never|change] # default: change
# on_failure: [always|never|change] # default: always
and as you can see in the log it just stalls after a few psql calls.
0.01s$ psql --version
psql (PostgreSQL) 9.3.5
before_install.2
0.02s$ psql -U postgres -c 'create database staging'
CREATE DATABASE
before_install.3
0.22s$ psql -U postgres -c 'create database rsgb'
CREATE DATABASE
before_install.4
1.04s$ psql -U postgres -d rsgb -c 'create extension postgis'
CREATE EXTENSION
$ psql -U postgres --list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
rsgb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
staging | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
travis | travis | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(6 rows)
(END)
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
I just spent about 3 hours troubleshooting this same issue and the problem is pretty simple once you understand why. psql is simply trying to page the output. There are multiple ways to disable the pager, but the solution I went with was to set the PAGER=cat environment variable in .travis.yml like so:
env:
- PGUSER=postgres
PAGER=cat
I have a postgresql (9.1.4) server running on a remote machine (Ubuntu 12.04), and I'm having trouble accessing it on my local machine. Specifically, on my local machine, I can access the remote server using only the username "postgres" and the database name "postgres", even though I have other user names and databases. First some of the background information. While in the interactive environment invoked with "psql" on the remote machine, I created another user, called "mxtxdb", and set the password for that user. I also created a database called "mxtxdb". To demonstrate their existence, I logged onto the remote machine and ran:
sudo su postgres
Password:
postgres#myhost:~$ psql
psql (9.1.4)
Type "help" for help.
postgres=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------+-----------
mxtxdb | | {}
postgres | Superuser, Create role, Create DB, Replication | {}
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
mxtxdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | mxtxdb=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Furthermore, while still on the remote machine, I can access the database from the "mxtxdb" postgresql account:
postgres#myhost:~$ psql -h localhost -U mxtxdb -d mxtxdb
Password for user mxtxdb:
psql (9.1.4)
Type "help" for help.
mxtxdb=>
But when I try to do the same on the local machine, I get this:
psql -h <server's address> -U postgres -d "postgres" -p 5432
Password for user postgres:
psql (9.1.4)
Type "help" for help.
postgres=#
and
psql -h <server's address> -U "postgres" -d "mxtxdb" -p 5432
Password for user postgres:
psql: FATAL: database "mxtxdb" does not exist
and
psql -h <server's address> -U "mxtxdb" -d "mxtxdb" -p 5432
Password for user mxtxdb:
psql: FATAL: role "mxtxdb" does not exist
The last two lines of my pg_hba.conf file are:
host all mxtxdb 0.0.0.0/0 trust
host all postgres 0.0.0.0/0 trust
and the listen_addresses is set to '*' (and uncommented) in the postgresql.conf file.
Any idea why I cannot access other accounts or databases remotely, but I can when ssh'd into the remote machine?
First of all, thanks to Craig Richard for your suggestions, and in fact you were both correct. Here's what happened: I installed PostgreSQL using sudo apt-get install postgresql-9.1, and then I followed the instructions for configuring the server at http://www.postgresql.org/docs/9.1/interactive/runtime.html . Unfortunately, what I did not notice was that the apt-get install process creates a postgresql database cluster by default, so when I followed the instructions to create a new one, I then had two clusters, one at /var/lib/postgresql/9.1/main and one at /usr/local/pgsql/data, and I was interacting with different clusters when I logged in locally vs remotely. When I removed the second cluster, the issue was resolved and I was able to connect remotely.
My takeaways from this process: when installing postgresql with apt-get, and then following the instructions in the postgresql manual, be aware that the install process will create a postgres operating system user by default, and will also create a database cluster by default, so those steps in the manual do not need to be performed. If anyone experiences an error like:
"FATAL: role/database '<name>' does not exist"
when you are sure you've created that user or database, check to ensure that you're operating with the correct database cluster.
Thanks again!