I have successfully setup replication for my Postgres database using pg_pool.
Then I stopped the master server and checked the pool status. It is as like below
postgres=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role
---------+------------+------+--------+-----------+--------
0 | 10.140.0.9 | 5432 | 3 | 0.500000 | slave
1 | 10.140.0.7 | 5432 | 2 | 0.500000 | master
(2 rows)
Then I started the server, but it still shows the same status for the slave. So I used the following command to start the node:
/usr/sbin/pcp_node_info 10 10.140.0.9 5432 postgres postgres 1
But it is giving "EOFError" error. Please help to solve this issue.
Or please let me know a way to bring back the status 2 from status 3?
I solved the issue myself. In configuration the pcp port is 9898. Also there should be no space before password in pcp.conf file.
The pcp command should be as follows
/usr/sbin/pcp_node_info 10 localhost 9898 postgres postgres 1
Related
I am using PostgreSQL 14 and ubuntu as my OS.
I have done the database replication in synchronous mode
Now I want to add another server and the relationship between primary and new_standby node will be asynchronous.
Another problem, how to set application_name for different nodes? I have got same application_name=14/main for my two standby servers
Can anyone help me to solve these issues?
Yes, you can run it simultaneously.
You should have this configuration in your primary node:
listen_addresses = '*'
port = 5432
wal_level = hot_standby
max_wal_senders = 16
wal_keep_segments = 32
synchronous_commit = on
synchronous_standby_names = 'pgsql_0_node_0'
Restart the node to take the changes:
$ systemctl restart postgresql-14
Create the replication role:
$ CREATE ROLE replication_user WITH LOGIN PASSWORD 'PASSWORD' REPLICATION;
And configure this in your standby nodes:
Both:
wal_level = hot_standby
hot_standby = on
Sync:
standby_mode = 'on'
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_0 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Async:
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_1 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Restart the node to take the changes:
$ systemctl restart postgresql-14
Then, you can run this in your Primary node to see the replication nodes:
$ SELECT pid,usename,application_name,state,sync_state FROM pg_stat_replication;
pid | usename | application_name | state | sync_state
-------+------------------+------------------+-----------+------------
10951 | replication_user | pgsql_0_node_1 | streaming | async
10952 | replication_user | pgsql_0_node_0 | streaming | sync
(2 rows)
I am using airflow 1.10.9 and celery worker. I have dags which run whenever task comes and it spins up new ec2 instance and it connects to RDS on the basis of logic but ec2 holds the connection even when there no task is running and it keeps holding connection until Auto scaling scales down the instance.
RDS Details -
Class : db.t3.xlarge
Engine : PostgreSQL
I have checked the RDS logs but no luck.
LOG: could not receive data from client: Connection reset by peer
here is RDS connections.
state | wait_event | wait_event_type | count
--------+---------------------+-----------------+-------
| AutoVacuumMain | Activity | 1
| BgWriterHibernate | Activity | 1
| CheckpointerMain | Activity | 1
idle | ClientRead | Client | 525
| LogicalLauncherMain | Activity | 1
| WalWriterMain | Activity | 1
active | | | 1
All the connections are from celery workers.
Any help is appreciated.
I have the following docker compose file:
version: "3"
services:
postgres:
image: postgres:11.2-alpine
environment:
POSTGRES_PASSWORD: root
POSTGRES_USER: root
ports:
- "5432:5432"
volumes:
- ./init-db/init-db.sql:/docker-entrypoint-initdb.d/init.sql
This is the init-db.sql:
CREATE TABLE users (
email VARCHAR(355) UNIQUE NOT NULL,
password VARCHAR(256) NOT NULL
);
CREATE TABLE products (
id SERIAL PRIMARY KEY,
title VARCHAR(100) NOT NULL,
price NUMERIC(6, 2) NOT NULL,
category INT NOT NULL
);
INSERT INTO users VALUES ('test#test.com', 'Test*123');
INSERT INTO products (title, price, category) VALUES ('Truco', 9.90, 13);
When I run docker-compose up, I'm getting this error:
server started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
/docker-entrypoint-initdb.d/init.sql: Permission denied
I already tried to:
chmod 777 on the sql file
chmod -x on the sql file
Run docker and docker-compose using sudo
Any idea?
I found the easy way to solve this...
You should use "build" way to create postgres service
And DO NOT setting the volume for init.sql, it will cause the permission problem.
postgres:
build: ./postgres
Create a Dockerfile for postgres like this
FROM postgres:12
COPY ./init.sql /docker-entrypoint-initdb.d/init.sql
CMD ["docker-entrypoint.sh", "postgres"]
Then it should works out.
Hope my answer would help you!
For me problem was in my machine. enabled SELinux access control, which did not allow for containers to expand files.
Solution, disable SELinux:
echo SELINUX=disabled > /etc/selinux/config
SELINUXTYPE=targeted >> /etc/selinux/config
setenforce 0
From this
I had a similar problem when using the ADD command.
When using ADD (eg to download a file) the default chmod value is 711. When using the COPY command the chmod will match the hosts chmod of the file where you copy it from. The solution is to set the permission before copying, or change them in your Dockerfile after they've been copied.
It looks like there will finally be a "COPY --chmod 775" flag available in the upcoming docker 20 which will make this easier.
https://github.com/moby/moby/issues/34819
When the sql file in /docker-entrypoint-initdb.d/ has a permission of 775 then the file is run correctly.
You can test this within the image (override entrypoint to /bin/bash) using:
docker-entrypoint.sh postgres
I have same problem on my macOS, but it's OK on my ubuntu laptop.
I found the problem is that MacOS cannot let docker access the folder.
Following link may resolve your problem.
https://stackoverflow.com/a/58482702/10752354
Besides, in my case, I should add one line code in my init.sql file because the default database is "root", and I should change "root" database into "postgres" database, or in your case for another custom database instead of root.
> docker-compose exec db psql -U root
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.
root=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privilege
s
-----------+-------+----------+------------+------------+-----------------
--
postgres | root | UTF8 | en_US.utf8 | en_US.utf8 |
root | root | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | root | UTF8 | en_US.utf8 | en_US.utf8 | =c/root
+
| | | | | root=CTc/root
template1 | root | UTF8 | en_US.utf8 | en_US.utf8 | =c/root
+
| | | | | root=CTc/root
(4 rows)
so I need to add \c postgres in my init.sql file:
\c postgres // for creating table in the database named 'postgres'
create table sample_table. ( ... )
I tried with the following compose file and it seems working as expected. are you sure form the path of the files you use?
version: "3"
services:
postgres:
image: postgres:11.2-alpine
environment:
POSTGRES_PASSWORD: root
POSTGRES_USER: root
ports:
- "5432:5432"
volumes:
- ./init-db.sql:/docker-entrypoint-initdb.d/init.sql
files structure
drwxr-xr-x 4 shihadeh 502596769 128B Feb 28 22:37 .
drwxr-xr-x 12 shihadeh 502596769 384B Feb 28 22:36 ..
-rw-r--r-- 1 shihadeh 502596769 244B Feb 28 22:37 docker-compose.yml
-rw-r--r-- 1 shihadeh 502596769 380B Feb 28 22:37 init-db.sql
output of docker-compose up
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... sh: locale: not found
postgres_1 | 2020-02-28 21:45:01.363 UTC [26] WARNING: no usable system locales were found
postgres_1 | ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2020-02-28 21:45:02.272 UTC [30] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-02-28 21:45:02.294 UTC [31] LOG: database system was shut down at 2020-02-28 21:45:01 UTC
postgres_1 | 2020-02-28 21:45:02.299 UTC [30] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
postgres_1 | CREATE TABLE
postgres_1 | CREATE TABLE
postgres_1 | INSERT 0 1
postgres_1 | INSERT 0 1
postgres_1 |
postgres_1 |
postgres_1 | waiting for server to shut down....2020-02-28 21:45:02.776 UTC [30] LOG: received fast shutdown request
postgres_1 | 2020-02-28 21:45:02.779 UTC [30] LOG: aborting any active transactions
postgres_1 | 2020-02-28 21:45:02.781 UTC [30] LOG: background worker "logical replication launcher" (PID 37) exited with exit code 1
postgres_1 | 2020-02-28 21:45:02.781 UTC [32] LOG: shutting down
postgres_1 | 2020-02-28 21:45:02.826 UTC [30] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2020-02-28 21:45:02.890 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-02-28 21:45:02.890 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-02-28 21:45:02.895 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-02-28 21:45:02.915 UTC [43] LOG: database system was shut down at 2020-02-28 21:45:02 UTC
postgres_1 | 2020-02-28 21:45:02.921 UTC [1] LOG: database system is ready to accept connections
Just adding my solution even when it is a little late:
My problem:
I am installing the source files from an udemy course:
https://github.com/rockthejvm/spark-essentials
We have 2 important things here:
Docker compose file: docker-compose.yml
Folder and sql file to do db initialization: ./sql/bd.sql
I downloaded the zip file to my Ubuntu 20.04
When doing docker-compose up, I got the error:
Attaching to postgres
postgres | ls: cannot open directory '/docker-entrypoint-initdb.d/': Permission denied
postgres exited with code 2
SOLUTION
Before running doccker-compose up do:
run chmod 777 ON FOLDER called sql that contains file db.sql
chmod 777 sql
Explanation:
The folder called sql contains file db.sql to init the database from inside the docker container so, we need to give the permissions to run the file inside folde called sql.
I think you only need to do as below.
Give permisions to your folder:
chmod 777 init-db
be sure your sql file can be read by all users:
chmod 666 init-db.sql
Postgres shuts down immediately when started with docker-compose. The yaml file used is below
version: '2'
services:
postgres:
image: postgres:9.5
container_name: local-postgres9.5
ports:
- "5432:5432"
The log when docker-compose up command is executed
Creating local-postgres9.5
Attaching to local-postgres9.5
local-postgres9.5 | The files belonging to this database system will be owned by user "postgres".
local-postgres9.5 | This user must also own the server process.
local-postgres9.5 |
local-postgres9.5 | The database cluster will be initialized with locale "en_US.utf8".
local-postgres9.5 | The default database encoding has accordingly been set to "UTF8".
local-postgres9.5 | The default text search configuration will be set to "english".
local-postgres9.5 |
local-postgres9.5 | Data page checksums are disabled.
local-postgres9.5 |
local-postgres9.5 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
local-postgres9.5 | creating subdirectories ... ok
local-postgres9.5 | selecting default max_connections ... 100
local-postgres9.5 | selecting default shared_buffers ... 128MB
local-postgres9.5 | selecting dynamic shared memory implementation ... posix
local-postgres9.5 | creating configuration files ... ok
local-postgres9.5 | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
local-postgres9.5 | initializing pg_authid ... ok
local-postgres9.5 | initializing dependencies ... ok
local-postgres9.5 | creating system views ... ok
local-postgres9.5 | loading system objects' descriptions ... ok
local-postgres9.5 | creating collations ... ok
local-postgres9.5 | creating conversions ... ok
local-postgres9.5 | creating dictionaries ... ok
local-postgres9.5 | setting privileges on built-in objects ... ok
local-postgres9.5 | creating information schema ... ok
local-postgres9.5 | loading PL/pgSQL server-side language ... ok
local-postgres9.5 | vacuuming database template1 ... ok
local-postgres9.5 | copying template1 to template0 ... ok
local-postgres9.5 | copying template1 to postgres ... ok
local-postgres9.5 | syncing data to disk ... ok
local-postgres9.5 |
local-postgres9.5 | WARNING: enabling "trust" authentication for local connections
local-postgres9.5 | You can change this by editing pg_hba.conf or using the option -A, or
local-postgres9.5 | --auth-local and --auth-host, the next time you run initdb.
local-postgres9.5 |
local-postgres9.5 | Success. You can now start the database server using:
local-postgres9.5 |
local-postgres9.5 | pg_ctl -D /var/lib/postgresql/data -l logfile start
local-postgres9.5 |
local-postgres9.5 | ****************************************************
local-postgres9.5 | WARNING: No password has been set for the database.
local-postgres9.5 | This will allow anyone with access to the
local-postgres9.5 | Postgres port to access your database. In
local-postgres9.5 | Docker's default configuration, this is
local-postgres9.5 | effectively any other container on the same
local-postgres9.5 | system.
local-postgres9.5 |
local-postgres9.5 | Use "-e POSTGRES_PASSWORD=password" to set
local-postgres9.5 | it in "docker run".
local-postgres9.5 | ****************************************************
local-postgres9.5 | waiting for server to start....LOG: database system was shut down at 2016-05-16 16:51:54 UTC
local-postgres9.5 | LOG: MultiXact member wraparound protections are now enabled
local-postgres9.5 | LOG: database system is ready to accept connections
local-postgres9.5 | LOG: autovacuum launcher started
local-postgres9.5 | done
local-postgres9.5 | server started
local-postgres9.5 | ALTER ROLE
local-postgres9.5 |
local-postgres9.5 |
local-postgres9.5 | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
local-postgres9.5 |
local-postgres9.5 | LOG: received fast shutdown request
local-postgres9.5 | LOG: aborting any active transactions
local-postgres9.5 | LOG: autovacuum launcher shutting down
local-postgres9.5 | LOG: shutting down
local-postgres9.5 | waiting for server to shut down....LOG: database system is shut down
local-postgres9.5 | done
local-postgres9.5 | server stopped
local-postgres9.5 |
local-postgres9.5 | PostgreSQL init process complete; ready for start up.
local-postgres9.5 |
local-postgres9.5 | LOG: database system was shut down at 2016-05-16 16:51:55 UTC
local-postgres9.5 | LOG: MultiXact member wraparound protections are now enabled
local-postgres9.5 | LOG: database system is ready to accept connections
local-postgres9.5 | LOG: autovacuum launcher started
Postgres seems to work fine when a container is started using the same image with docker run
docker run --name local-postgres9.5 -p 5432:5432 postgres:9.5
If you look at your log output, the following lines appear towards the end:
local-postgres9.5 | server stopped
local-postgres9.5 |
local-postgres9.5 | PostgreSQL init process complete; ready for start up.
Apparently, stopping and restarting the Postgres server is part of the initialisation process. In fact, the second-to-last line says
local-postgres9.5 | LOG: database system is ready to accept connections.
I'm using the same Postgres docker image version of yours (9.5) and I was running into the same problem.
The first time the container is created, Postgres run a series of commands and in the end, it just sends a shutdown signal and the server becomes unresponsive (but it works the second time onwards).
After several trials, I figured that once I tried to connect to the server, before it was ready to accept connections, PostgresDB would shutdown unexpectedly, and the same would happen for any client trying to connect remotely (outside the container).
I came across this error in the following scenario - I have two containers: one for the PostgresDB itself and other containing a Postgres client (psql) that tries to connect to the first container to create some users, databases and run a liquibase script that creates all the schemas.
At first, I was using a loop in my bash script checking if the server was available every 5 seconds, then after the first attempt, Postgres issued a signal to shutdown and becomes unresponsive.
I could get rid of this error by adding a healthcheck in my docker-compose.yml:
Checkout the CHANGE 1 and CHANGE 2 comments below
version: '3.9'
services:
postgres-db:
container_name: ${POSTGRES_HOST}
image: postgres:9.5
restart: always
ports:
- "5432:5432"
command: postgres
expose:
- 5432
environment:
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
POSTGRES_DB: "${POSTGRES_DB}"
#this ENV variable is only required for the healthcheck section - if you don't specify it, the check command will fail stating the root user doesn't exist in posgres
PGUSER: "postgres"
healthcheck:
#CHANGE 1: this command checks if the database is ready, right on the source db server
test: [ "CMD-SHELL", "pg_isready" ]
interval: 5s
timeout: 5s
retries: 5
liquibase:
container_name: liquibase-schema-config
image: company/liquibase
build:
context: ./liquibase
environment:
- PGPASSWORD=${POSTGRES_PASSWORD}
- PGPORT=${POSTGRES_PORT}
- PGHOST=${POSTGRES_HOST}
- PGUSER=${POSTGRES_USER}
- PGDATABASE=${POSTGRES_DB}
- JDBC_URL=jdbc:postgresql://${POSTGRES_HOST}:${POSTGRES_PORT}/
- LIQUIBASE_HOME=${LIQUIBASE_HOME}
depends_on:
#CHANGE 2: it prevents issuing a request while the server is starting to depend on the healthy status of postgres-db
postgres-db:
condition: service_healthy
EDIT: There was another issue preventing Docker from accessing VPN protected sites (and even internet sites) while using Cisco AnyConnect on Linux (it doesn't happen on MacOS). To get around this unwanted behavior I installed openconnect sudo apt install openconnect on my Ubuntu and stopped using AnyConnect.
Hope it helps!
I tried your docker-compose and the service seems running in the container:
root#0afe99de0f0b:/# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
postgres 1 0.5 0.8 227148 16128 ? Ss 14:42 0:00 postgres
postgres 74 0.0 0.0 227148 1768 ? Ss 14:42 0:00 postgres: checkpointer process
postgres 75 0.0 0.0 227148 1772 ? Ss 14:42 0:00 postgres: writer process
postgres 76 0.0 0.0 227148 1768 ? Ss 14:42 0:00 postgres: wal writer process
postgres 77 0.0 0.1 227576 2720 ? Ss 14:42 0:00 postgres: autovacuum launcher process
postgres 78 0.0 0.0 82132 1888 ? Ss 14:42 0:00 postgres: stats collector process
root 79 2.0 0.0 21820 1984 ? Ss 14:42 0:00 /bin/bash
root 84 0.0 0.0 19092 1296 ? R+ 14:42 0:00 ps aux
Anyway, for my project I use another image for postgresql: https://github.com/sameersbn/docker-postgresql. This one works fine.
Add password under postgres service in docker-compose.yml as given in the screenshot. Thank you.
click to see the screenshot
version: '3'
services:
postgres:
image: postgres
environment:
- POSTGRES_PASSWORD=postgres_password
thanks for your help.
I failed to build a Postgres-XL cluster.
My gtm.confg :
listen_addresses = '0.0.0.0'
port = 6668
at 10.80.19.47.
My gtm_proxy.confg:
listen_addresses = '0.0.0.0'
port = 6666
gtm_host = '10.80.19.47'
gtm_port = 6668
after start gtm and gtm_proxy ,i show net information use lsof -i :
gtm 11730 postgres 6u IPv4 355914 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44017 (ESTABLISHED
gtm 11730 postgres 7u IPv4 355915 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44018 (ESTABLISHED)
gtm 11730 postgres 8u IPv4 355916 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44019 (ESTABLISHED)
Then i configure datanode like:
listen_addresses = '0.0.0.0'
port = 5432
pooler_port = 20002
gtm_host = '127.0.0.1'
gtm_port = 6666
and coordinator like:
listen_addresses = '0.0.0.0'
port = 3456
pooler_port = 20003
gtm_host = '127.0.0.1'
gtm_port = 6666
others are default ,then i start datanode with :
/usr/postgres-xl-9.2/bin/postgres --datanode -D /data1/pgsql/data &
start coordinator with:
/usr/postgres-xl-9.2/bin/postgres --coordinator -D /data1/pgsql/coordinator &
when i connect to Postgres-XL use coordinator with :
psql -h 127.0.0.1 -p 3456 postgres;
when i want to create a database with:
create database test;
but failed,it shows :
No Datanode defined in cluster
You need to define at least 1 Datanode with CREATE NODE.
then i check the cluster with :
psql -h 127.0.0.1 -p 3456 postgres -c "select * from pgxc_node order by 1";
it shows just one node like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
-------------------+-----------+-----------+-----------+----------------+------------------+-----------
coordinator_node2 | C | 5432 | localhost | f | f | 738118815
and i connect to port 5432 it shows the similar information like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
------------+-----------+-----------+-----------+----------------+------------------+------------
data_node2 | C | 5432 | localhost | f | f | -923817565
what wrong with my Configuration?
Thanks all.
You need to define datanode something like this.
./psql -c "CREATE NODE datanode1 WITH (TYPE = 'datanode', PORT = 7777)" postgres -p 3456 -h 127.0.0.1
I'm answering this question a year after. However, I'd suggest you to use pgxc_ctl command line tool as suggested by the Postgres XL developers themselves. It will keep you away from the hectic tasks of manually configuring the cluster.
You can refer to the official documentation of Postgres XL for this. After you successfully installed pgxc_ctl command line tool, type in
prepare config minimal
in order to generate a basic configuration file to configure the cluster. Edit your changes in pgxc_ctl/pgxc_ctl.conf accordingly. Type in exit and finally you can initialize all the clusters if your configuration is okay. For this, type in
pgxc_ctl init all