I'm trying to create a postgresql docker container using docker-compose, but it seems that when I use a pre-ready postgresql image the console never return to the host, and if I hit Ctrl+C the container ends up being stoped. Here is what I got:
docker-compose.yml
version: '2'
services:
database:
container_name: database_server
build:
context: .
dockerfile: Dockerfile_database
image: database:tag
expose:
- "5432"
Dockerfile_database
FROM library/postgres
ADD init.sql /docker-entrypoint-initdb.d/
init.sql
CREATE USER usuario WITH PASSWORD 'senha';
CREATE DATABASE telesaude;
GRANT ALL PRIVILEGES ON DATABASE telesaude TO usuario;
and during the container creation the console shows some postgresql logs, such as this:
Building database
Step 1/2 : FROM library/postgres
---> 4023a747a01a
Step 2/2 : ADD init.sql /docker-entrypoint-initdb.d/
---> e264774d1f0b
Removing intermediate container 8df766b5a9f2
Successfully built e264774d1f0b
WARNING: Image for service database was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating database_server
Attaching to database_server
database_server | The files belonging to this database system will be owned by user "postgres".
database_server | This user must also own the server process.
database_server |
database_server | The database cluster will be initialized with locale "en_US.utf8".
database_server | The default database encoding has accordingly been set to "UTF8".
database_server | The default text search configuration will be set to "english".
database_server |
database_server | Data page checksums are disabled.
database_server |
database_server | fixing permissions on existing directory /var/lib/postgresql/data ... ok
database_server | creating subdirectories ... ok
database_server | selecting default max_connections ... 100
database_server | selecting default shared_buffers ... 128MB
database_server | selecting dynamic shared memory implementation ... posix
database_server | creating configuration files ... ok
database_server | running bootstrap script ... ok
database_server | performing post-bootstrap initialization ... ok
database_server | syncing data to disk ...
database_server | WARNING: enabling "trust" authentication for local connections
database_server | You can change this by editing pg_hba.conf or using the option -A, or
database_server | --auth-local and --auth-host, the next time you run initdb.
database_server | ok
database_server |
database_server | Success. You can now start the database server using:
database_server |
database_server | pg_ctl -D /var/lib/postgresql/data -l logfile start
database_server |
database_server | ****************************************************
database_server | WARNING: No password has been set for the database.
database_server | This will allow anyone with access to the
database_server | Postgres port to access your database. In
database_server | Docker's default configuration, this is
database_server | effectively any other container on the same
database_server | system.
database_server |
database_server | Use "-e POSTGRES_PASSWORD=password" to set
database_server | it in "docker run".
database_server | ****************************************************
database_server | waiting for server to start....LOG: database system was shut down at 2017-02-02 21:15:46 UTC
database_server | LOG: MultiXact member wraparound protections are now enabled
database_server | LOG: database system is ready to accept connections
database_server | LOG: autovacuum launcher started
database_server | done
database_server | server started
database_server | ALTER ROLE
database_server |
database_server |
database_server | /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
database_server | CREATE ROLE
database_server | CREATE DATABASE
database_server | GRANT
database_server |
database_server |
database_server | LOG: received fast shutdown request
database_server | LOG: aborting any active transactions
database_server | waiting for server to shut down...LOG: autovacuum launcher shutting down
database_server | .LOG: shutting down
database_server | LOG: database system is shut down
database_server | done
database_server | server stopped
database_server |
database_server | PostgreSQL init process complete; ready for start up.
database_server |
database_server | LOG: database system was shut down at 2017-02-02 21:15:49 UTC
database_server | LOG: MultiXact member wraparound protections are now enabled
database_server | LOG: database system is ready to accept connections
database_server | LOG: autovacuum launcher started
(console never returns to host after this)
What should I do so the container keeps running/started and the console returns to host? I have to use docker-compose. Can't use docker run -d ...
To run the container in the background with compose, use the detach option:
docker-compose up -d
Related
I am running very basic docker-compose with following docker-compose.yml and getting following error.
docker-compose.yml
services:
redis:
image: redis
db:
image: postgres:9.4
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
vote:
image: voting-app
ports:
- 5000:80
depends_on:
- redis
worker:
image: worker-app
depends_on:
- redis
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
result:
image: result-app
ports:
- 5001:80
depends_on:
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
Error generated,
db_1 | vacuuming database template1 ... ok
db_1 | copying template1 to template0 ... ok
db_1 | copying template1 to postgres ... ok
db_1 | syncing data to disk ...
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | postgres -D /var/lib/postgresql/data
db_1 | or
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2021-03-03 12:45:13 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | waiting for server to shut down....LOG: shutting down
db_1 | LOG: database system is shut down
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2021-03-03 12:45:15 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
I checked on various stackoverflow existing questions but none of them are rectifying actual issue. Some suggesting to run postgres docker and bash it to update the password but its a workaround. Anyone has any concrete solution to this?
I figured it out finally and would be good to share what mistake I had done.
I changed the database configuration by providing environment variable in the docker-compose.yml file in db service as mentioned in the question already,
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
But the same needs to be updated in all the other custom services (open source github) which are trying to connect to this depende-on postgres database with same user/password/database which was not done.
However the error given by docker-compose is not explaining the situation clearly.
In All, I updated the applications with above username/password/database in connection property and the containers are up.
I have the following docker compose file:
version: "3"
services:
postgres:
image: postgres:11.2-alpine
environment:
POSTGRES_PASSWORD: root
POSTGRES_USER: root
ports:
- "5432:5432"
volumes:
- ./init-db/init-db.sql:/docker-entrypoint-initdb.d/init.sql
This is the init-db.sql:
CREATE TABLE users (
email VARCHAR(355) UNIQUE NOT NULL,
password VARCHAR(256) NOT NULL
);
CREATE TABLE products (
id SERIAL PRIMARY KEY,
title VARCHAR(100) NOT NULL,
price NUMERIC(6, 2) NOT NULL,
category INT NOT NULL
);
INSERT INTO users VALUES ('test#test.com', 'Test*123');
INSERT INTO products (title, price, category) VALUES ('Truco', 9.90, 13);
When I run docker-compose up, I'm getting this error:
server started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
/docker-entrypoint-initdb.d/init.sql: Permission denied
I already tried to:
chmod 777 on the sql file
chmod -x on the sql file
Run docker and docker-compose using sudo
Any idea?
I found the easy way to solve this...
You should use "build" way to create postgres service
And DO NOT setting the volume for init.sql, it will cause the permission problem.
postgres:
build: ./postgres
Create a Dockerfile for postgres like this
FROM postgres:12
COPY ./init.sql /docker-entrypoint-initdb.d/init.sql
CMD ["docker-entrypoint.sh", "postgres"]
Then it should works out.
Hope my answer would help you!
For me problem was in my machine. enabled SELinux access control, which did not allow for containers to expand files.
Solution, disable SELinux:
echo SELINUX=disabled > /etc/selinux/config
SELINUXTYPE=targeted >> /etc/selinux/config
setenforce 0
From this
I had a similar problem when using the ADD command.
When using ADD (eg to download a file) the default chmod value is 711. When using the COPY command the chmod will match the hosts chmod of the file where you copy it from. The solution is to set the permission before copying, or change them in your Dockerfile after they've been copied.
It looks like there will finally be a "COPY --chmod 775" flag available in the upcoming docker 20 which will make this easier.
https://github.com/moby/moby/issues/34819
When the sql file in /docker-entrypoint-initdb.d/ has a permission of 775 then the file is run correctly.
You can test this within the image (override entrypoint to /bin/bash) using:
docker-entrypoint.sh postgres
I have same problem on my macOS, but it's OK on my ubuntu laptop.
I found the problem is that MacOS cannot let docker access the folder.
Following link may resolve your problem.
https://stackoverflow.com/a/58482702/10752354
Besides, in my case, I should add one line code in my init.sql file because the default database is "root", and I should change "root" database into "postgres" database, or in your case for another custom database instead of root.
> docker-compose exec db psql -U root
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.
root=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privilege
s
-----------+-------+----------+------------+------------+-----------------
--
postgres | root | UTF8 | en_US.utf8 | en_US.utf8 |
root | root | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | root | UTF8 | en_US.utf8 | en_US.utf8 | =c/root
+
| | | | | root=CTc/root
template1 | root | UTF8 | en_US.utf8 | en_US.utf8 | =c/root
+
| | | | | root=CTc/root
(4 rows)
so I need to add \c postgres in my init.sql file:
\c postgres // for creating table in the database named 'postgres'
create table sample_table. ( ... )
I tried with the following compose file and it seems working as expected. are you sure form the path of the files you use?
version: "3"
services:
postgres:
image: postgres:11.2-alpine
environment:
POSTGRES_PASSWORD: root
POSTGRES_USER: root
ports:
- "5432:5432"
volumes:
- ./init-db.sql:/docker-entrypoint-initdb.d/init.sql
files structure
drwxr-xr-x 4 shihadeh 502596769 128B Feb 28 22:37 .
drwxr-xr-x 12 shihadeh 502596769 384B Feb 28 22:36 ..
-rw-r--r-- 1 shihadeh 502596769 244B Feb 28 22:37 docker-compose.yml
-rw-r--r-- 1 shihadeh 502596769 380B Feb 28 22:37 init-db.sql
output of docker-compose up
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... sh: locale: not found
postgres_1 | 2020-02-28 21:45:01.363 UTC [26] WARNING: no usable system locales were found
postgres_1 | ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2020-02-28 21:45:02.272 UTC [30] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-02-28 21:45:02.294 UTC [31] LOG: database system was shut down at 2020-02-28 21:45:01 UTC
postgres_1 | 2020-02-28 21:45:02.299 UTC [30] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
postgres_1 | CREATE TABLE
postgres_1 | CREATE TABLE
postgres_1 | INSERT 0 1
postgres_1 | INSERT 0 1
postgres_1 |
postgres_1 |
postgres_1 | waiting for server to shut down....2020-02-28 21:45:02.776 UTC [30] LOG: received fast shutdown request
postgres_1 | 2020-02-28 21:45:02.779 UTC [30] LOG: aborting any active transactions
postgres_1 | 2020-02-28 21:45:02.781 UTC [30] LOG: background worker "logical replication launcher" (PID 37) exited with exit code 1
postgres_1 | 2020-02-28 21:45:02.781 UTC [32] LOG: shutting down
postgres_1 | 2020-02-28 21:45:02.826 UTC [30] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2020-02-28 21:45:02.890 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-02-28 21:45:02.890 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-02-28 21:45:02.895 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-02-28 21:45:02.915 UTC [43] LOG: database system was shut down at 2020-02-28 21:45:02 UTC
postgres_1 | 2020-02-28 21:45:02.921 UTC [1] LOG: database system is ready to accept connections
Just adding my solution even when it is a little late:
My problem:
I am installing the source files from an udemy course:
https://github.com/rockthejvm/spark-essentials
We have 2 important things here:
Docker compose file: docker-compose.yml
Folder and sql file to do db initialization: ./sql/bd.sql
I downloaded the zip file to my Ubuntu 20.04
When doing docker-compose up, I got the error:
Attaching to postgres
postgres | ls: cannot open directory '/docker-entrypoint-initdb.d/': Permission denied
postgres exited with code 2
SOLUTION
Before running doccker-compose up do:
run chmod 777 ON FOLDER called sql that contains file db.sql
chmod 777 sql
Explanation:
The folder called sql contains file db.sql to init the database from inside the docker container so, we need to give the permissions to run the file inside folde called sql.
I think you only need to do as below.
Give permisions to your folder:
chmod 777 init-db
be sure your sql file can be read by all users:
chmod 666 init-db.sql
In one folder I have 4 files: base.py, Dockerfile, docker-compose.yml and wait-for-it.sh.
base.py:
import psycopg2
print("JJJJJJJJ")
conn = psycopg2.connect("dbname='base123' user='postgres' host='db' password='pw1234'")
cur = conn.cursor()
cur.execute("CREATE TABLE test(id serial PRIMARY KEY, num int);")
Dockerfile:
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -y install python-pip
RUN apt-get update
RUN pip install --upgrade pip
RUN pip install psycopg2-binary
COPY base.py base.py
COPY wait-for-it.sh wait-for-it.sh
RUN chmod +x /wait-for-it.sh
CMD ["python","base.py"]
docker-compose.yml
version: '3'
services:
db:
image: 'postgres:latest'
expose:
- "5432"
ports:
- "5559:5432"
environment:
POSTGRES_PASSWORD: pw1234
POSTGRES_DB: base123
aprrka:
build: .
depends_on:
- db
command: ["./wait-for-it.sh", "db:5432", "--", "python", "base.py"]
wait-fot-it.sh:
link to code in github
After docker-compose up I have this OUTPUT:
...
Creating postgres_db_1 ... done
Creating postgres_aprrka_1 ... done
Attaching to postgres_db_1, postgres_aprrka_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
aprrka_1 | wait-for-it.sh: waiting 15 seconds for db:5432
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....2018-08-10 09:59:13.529 UTC [38] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:13.582 UTC [39] LOG: database system was shut down at 2018-08-10 09:59:12 UTC
db_1 | 2018-08-10 09:59:13.599 UTC [38] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | 2018-08-10 09:59:14.585 UTC [38] LOG: received fast shutdown request
db_1 | waiting for server to shut down....2018-08-10 09:59:14.593 UTC [38] LOG: aborting any active transactions
db_1 | 2018-08-10 09:59:14.613 UTC [38] LOG: worker process: logical replication launcher (PID 45) exited with exit code 1
db_1 | 2018-08-10 09:59:14.613 UTC [40] LOG: shutting down
db_1 | 2018-08-10 09:59:14.660 UTC [38] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2018-08-10 09:59:14.729 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:14.806 UTC [65] LOG: database system was shut down at 2018-08-10 09:59:14 UTC
db_1 | 2018-08-10 09:59:14.829 UTC [1] LOG: database system is ready to accept connections
db_1 | 2018-08-10 09:59:15.217 UTC [72] LOG: incomplete startup packet
aprrka_1 | wait-for-it.sh: db:5432 is available after 5 seconds
aprrka_1 | JJJJJJJJ
postgres_aprrka_1 exited with code 0
When I type command: psql -U postgres -h localhost -p 5559 in another terminal I can connect to base1234, but when type `\dt' i have no any relations, so my py file don't create table in postgres database. I think that no connection exist with postgres db. Please, help me
You need to close the cursor and commit all pending transactions to the database for the changes to be effective.
Add the following 2 lines at the end of your base.py:
cur.close()
conn.commit()
I'm running a Scrapy project with an Item Pipeline which writes data to a PostgreSQL database. It is a Docker Compose multi-container application with the following docker-compose.yml:
version: '3'
services:
privoxy:
build: ./privoxy
links:
- tor
tor:
build:
context: ./tor
args:
password: "foo"
scraper:
build: ./scraper
environment:
- http_proxy=http://privoxy:8118
- PGPASSWORD=iperpassword
links:
- tor
- privoxy
- db
volumes:
- scraper-data:/scraper/apkmirror_scraper/data
- scraper-jobdir:/scraper/apkmirror_scraper/crawls
db:
image: postgres
environment:
- POSTGRES_PASSWORD=iperpassword
volumes:
- db-data:/var/lib/postgresql/data
volumes:
scraper-data:
scraper-jobdir:
db-data:
I'm using the postgres image from the official repository (https://hub.docker.com/_/postgres/), and have made the /var/lib/postgresql/data directory into a volume following https://docs.docker.com/compose/compose-file/#volume-configuration-reference.
The problem is that when I start the application, the db service outputs logs and errors pertaining to things I tried previously, not what I'm trying to do now. This is what I see:
apkmirrorscrapercompose_tor_1 is up-to-date
apkmirrorscrapercompose_db_1 is up-to-date
apkmirrorscrapercompose_privoxy_1 is up-to-date
Recreating apkmirrorscrapercompose_scraper_1
Attaching to apkmirrorscrapercompose_tor_1, apkmirrorscrapercompose_db_1, apkmirrorscrapercompose_privoxy_1, apkmirrorscrapercompose_scraper_1
scraper_1 | List of databases
scraper_1 | Name | Owner | Encoding | Collate | Ctype | Access privileges
scraper_1 | -----------+----------+----------+------------+------------+-----------------------
scraper_1 | postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
scraper_1 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | (3 rows)
scraper_1 |
scraper_1 | Postgres is up - executing command
scraper_1 | Tor appears to be working. Proceeding with command...
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: apkmirror_scraper)
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'apkmirror_scraper', 'NEWSPIDER_MODULE': 'apkmirror_scraper.spiders', 'SPIDER_MODULES': ['apkmirror_scraper.spiders']}
scraper_1 | 2017-06-30 12:32:59 [scrapy.middleware] INFO: Enabled extensions:
scraper_1 | ['scrapy.extensions.corestats.CoreStats',
scraper_1 | 'scrapy.extensions.telnet.TelnetConsole',
scraper_1 | 'scrapy.extensions.memusage.MemoryUsage',
scraper_1 | 'scrapy.extensions.closespider.CloseSpider',
scraper_1 | 'scrapy.extensions.logstats.LogStats']
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2017-06-29 13:40:40 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | done
db_1 | server started
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | waiting for server to shut down....LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2017-06-29 13:40:41 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:41:53 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:43:22 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:52:02 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | ERROR: column "developer" of relation "apks" does not exist at character 31
db_1 | STATEMENT: INSERT INTO apks (url, title, developer, app, version_name, version_code, architectures, package, apk_file_size, android_min_version, android_target_version, supported_dpis, md5_signature, time_uploaded, time_scraped, image_urls, images, file_urls, files, created_on, updated_on) VALUES ('http://www.apkmirror.com/apk/sony-mobile-communications/backup-restore-unknown-developer/backup-restore-unknown-developer-1-2-a-1-23-release/backup-restore-1-2-a-1-23-android-apk-download/', 'Backup & restore 1.2.A.1.23', 'Sony Mobile Communications', 'Backup & restore', '1.2.A.1.23', '2360343', ARRAY['arm'], 'com.sonymobile.synchub', 9541452, '6.0', '7.1', ARRAY['nodpi'], 'eda39fedfc33ea939a5bb4926b3980c1', '2017-04-25T16:30:00+00:00'::timestamptz, '2017-06-29T14:10:39+00:00'::timestamptz, ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/ap_resize/ap_resize.php?src=http%3A%2F%2Fwww.apkmirror.com%2Fwp-content%2Fuploads%2F2017%2F04%2F58ff36aea8e0c.png&w=96&h=96&q=100'], '{}', ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/download.php?id=203068'], '{}', '2017-06-29T14:19:21.661193'::timestamp, '2017-06-29T14:19:21.661207'::timestamp) RETURNING apks.id
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 16:28:54 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
Although I'm not showing details on the application, my main point is that I'm positive that data shown in the error message is from a previous run, and not from the APK download page that I'm currently trying to scrape.
In short, it seems like PostgreSQL is 'caching' previous statements and trying to re-run them even after I stop the application, do docker-compose build and docker-compose up. It seems to me that this behavior must somehow be driven by the data in /var/lib/postgresql/data, since this volume is the only thing that is persisted across restarting the container.
Is this indeed the case? If so, how can I disable this 'statement caching' behavior?
This just means that docker-compose is re-starting your previously stopped container (or re-attaching to it, if it is already running), and because of there is no need to re-build the image:
apkmirrorscrapercompose_db_1 is up-to-date
So, what your are seeing "again" in the logs is the docker logs of that container which includes all the life-generated logs of postgres.
Simply remove the container and start again (the db data is persisted in the volume, so it will survive):
docker-compose rm db
Postgres shuts down immediately when started with docker-compose. The yaml file used is below
version: '2'
services:
postgres:
image: postgres:9.5
container_name: local-postgres9.5
ports:
- "5432:5432"
The log when docker-compose up command is executed
Creating local-postgres9.5
Attaching to local-postgres9.5
local-postgres9.5 | The files belonging to this database system will be owned by user "postgres".
local-postgres9.5 | This user must also own the server process.
local-postgres9.5 |
local-postgres9.5 | The database cluster will be initialized with locale "en_US.utf8".
local-postgres9.5 | The default database encoding has accordingly been set to "UTF8".
local-postgres9.5 | The default text search configuration will be set to "english".
local-postgres9.5 |
local-postgres9.5 | Data page checksums are disabled.
local-postgres9.5 |
local-postgres9.5 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
local-postgres9.5 | creating subdirectories ... ok
local-postgres9.5 | selecting default max_connections ... 100
local-postgres9.5 | selecting default shared_buffers ... 128MB
local-postgres9.5 | selecting dynamic shared memory implementation ... posix
local-postgres9.5 | creating configuration files ... ok
local-postgres9.5 | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
local-postgres9.5 | initializing pg_authid ... ok
local-postgres9.5 | initializing dependencies ... ok
local-postgres9.5 | creating system views ... ok
local-postgres9.5 | loading system objects' descriptions ... ok
local-postgres9.5 | creating collations ... ok
local-postgres9.5 | creating conversions ... ok
local-postgres9.5 | creating dictionaries ... ok
local-postgres9.5 | setting privileges on built-in objects ... ok
local-postgres9.5 | creating information schema ... ok
local-postgres9.5 | loading PL/pgSQL server-side language ... ok
local-postgres9.5 | vacuuming database template1 ... ok
local-postgres9.5 | copying template1 to template0 ... ok
local-postgres9.5 | copying template1 to postgres ... ok
local-postgres9.5 | syncing data to disk ... ok
local-postgres9.5 |
local-postgres9.5 | WARNING: enabling "trust" authentication for local connections
local-postgres9.5 | You can change this by editing pg_hba.conf or using the option -A, or
local-postgres9.5 | --auth-local and --auth-host, the next time you run initdb.
local-postgres9.5 |
local-postgres9.5 | Success. You can now start the database server using:
local-postgres9.5 |
local-postgres9.5 | pg_ctl -D /var/lib/postgresql/data -l logfile start
local-postgres9.5 |
local-postgres9.5 | ****************************************************
local-postgres9.5 | WARNING: No password has been set for the database.
local-postgres9.5 | This will allow anyone with access to the
local-postgres9.5 | Postgres port to access your database. In
local-postgres9.5 | Docker's default configuration, this is
local-postgres9.5 | effectively any other container on the same
local-postgres9.5 | system.
local-postgres9.5 |
local-postgres9.5 | Use "-e POSTGRES_PASSWORD=password" to set
local-postgres9.5 | it in "docker run".
local-postgres9.5 | ****************************************************
local-postgres9.5 | waiting for server to start....LOG: database system was shut down at 2016-05-16 16:51:54 UTC
local-postgres9.5 | LOG: MultiXact member wraparound protections are now enabled
local-postgres9.5 | LOG: database system is ready to accept connections
local-postgres9.5 | LOG: autovacuum launcher started
local-postgres9.5 | done
local-postgres9.5 | server started
local-postgres9.5 | ALTER ROLE
local-postgres9.5 |
local-postgres9.5 |
local-postgres9.5 | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
local-postgres9.5 |
local-postgres9.5 | LOG: received fast shutdown request
local-postgres9.5 | LOG: aborting any active transactions
local-postgres9.5 | LOG: autovacuum launcher shutting down
local-postgres9.5 | LOG: shutting down
local-postgres9.5 | waiting for server to shut down....LOG: database system is shut down
local-postgres9.5 | done
local-postgres9.5 | server stopped
local-postgres9.5 |
local-postgres9.5 | PostgreSQL init process complete; ready for start up.
local-postgres9.5 |
local-postgres9.5 | LOG: database system was shut down at 2016-05-16 16:51:55 UTC
local-postgres9.5 | LOG: MultiXact member wraparound protections are now enabled
local-postgres9.5 | LOG: database system is ready to accept connections
local-postgres9.5 | LOG: autovacuum launcher started
Postgres seems to work fine when a container is started using the same image with docker run
docker run --name local-postgres9.5 -p 5432:5432 postgres:9.5
If you look at your log output, the following lines appear towards the end:
local-postgres9.5 | server stopped
local-postgres9.5 |
local-postgres9.5 | PostgreSQL init process complete; ready for start up.
Apparently, stopping and restarting the Postgres server is part of the initialisation process. In fact, the second-to-last line says
local-postgres9.5 | LOG: database system is ready to accept connections.
I'm using the same Postgres docker image version of yours (9.5) and I was running into the same problem.
The first time the container is created, Postgres run a series of commands and in the end, it just sends a shutdown signal and the server becomes unresponsive (but it works the second time onwards).
After several trials, I figured that once I tried to connect to the server, before it was ready to accept connections, PostgresDB would shutdown unexpectedly, and the same would happen for any client trying to connect remotely (outside the container).
I came across this error in the following scenario - I have two containers: one for the PostgresDB itself and other containing a Postgres client (psql) that tries to connect to the first container to create some users, databases and run a liquibase script that creates all the schemas.
At first, I was using a loop in my bash script checking if the server was available every 5 seconds, then after the first attempt, Postgres issued a signal to shutdown and becomes unresponsive.
I could get rid of this error by adding a healthcheck in my docker-compose.yml:
Checkout the CHANGE 1 and CHANGE 2 comments below
version: '3.9'
services:
postgres-db:
container_name: ${POSTGRES_HOST}
image: postgres:9.5
restart: always
ports:
- "5432:5432"
command: postgres
expose:
- 5432
environment:
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
POSTGRES_DB: "${POSTGRES_DB}"
#this ENV variable is only required for the healthcheck section - if you don't specify it, the check command will fail stating the root user doesn't exist in posgres
PGUSER: "postgres"
healthcheck:
#CHANGE 1: this command checks if the database is ready, right on the source db server
test: [ "CMD-SHELL", "pg_isready" ]
interval: 5s
timeout: 5s
retries: 5
liquibase:
container_name: liquibase-schema-config
image: company/liquibase
build:
context: ./liquibase
environment:
- PGPASSWORD=${POSTGRES_PASSWORD}
- PGPORT=${POSTGRES_PORT}
- PGHOST=${POSTGRES_HOST}
- PGUSER=${POSTGRES_USER}
- PGDATABASE=${POSTGRES_DB}
- JDBC_URL=jdbc:postgresql://${POSTGRES_HOST}:${POSTGRES_PORT}/
- LIQUIBASE_HOME=${LIQUIBASE_HOME}
depends_on:
#CHANGE 2: it prevents issuing a request while the server is starting to depend on the healthy status of postgres-db
postgres-db:
condition: service_healthy
EDIT: There was another issue preventing Docker from accessing VPN protected sites (and even internet sites) while using Cisco AnyConnect on Linux (it doesn't happen on MacOS). To get around this unwanted behavior I installed openconnect sudo apt install openconnect on my Ubuntu and stopped using AnyConnect.
Hope it helps!
I tried your docker-compose and the service seems running in the container:
root#0afe99de0f0b:/# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
postgres 1 0.5 0.8 227148 16128 ? Ss 14:42 0:00 postgres
postgres 74 0.0 0.0 227148 1768 ? Ss 14:42 0:00 postgres: checkpointer process
postgres 75 0.0 0.0 227148 1772 ? Ss 14:42 0:00 postgres: writer process
postgres 76 0.0 0.0 227148 1768 ? Ss 14:42 0:00 postgres: wal writer process
postgres 77 0.0 0.1 227576 2720 ? Ss 14:42 0:00 postgres: autovacuum launcher process
postgres 78 0.0 0.0 82132 1888 ? Ss 14:42 0:00 postgres: stats collector process
root 79 2.0 0.0 21820 1984 ? Ss 14:42 0:00 /bin/bash
root 84 0.0 0.0 19092 1296 ? R+ 14:42 0:00 ps aux
Anyway, for my project I use another image for postgresql: https://github.com/sameersbn/docker-postgresql. This one works fine.
Add password under postgres service in docker-compose.yml as given in the screenshot. Thank you.
click to see the screenshot
version: '3'
services:
postgres:
image: postgres
environment:
- POSTGRES_PASSWORD=postgres_password