About coordinator and datanode in Postgres-XL - postgresql

thanks for your help.
I failed to build a Postgres-XL cluster.
My gtm.confg :
listen_addresses = '0.0.0.0'
port = 6668
at 10.80.19.47.
My gtm_proxy.confg:
listen_addresses = '0.0.0.0'
port = 6666
gtm_host = '10.80.19.47'
gtm_port = 6668
after start gtm and gtm_proxy ,i show net information use lsof -i :
gtm 11730 postgres 6u IPv4 355914 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44017 (ESTABLISHED
gtm 11730 postgres 7u IPv4 355915 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44018 (ESTABLISHED)
gtm 11730 postgres 8u IPv4 355916 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44019 (ESTABLISHED)
Then i configure datanode like:
listen_addresses = '0.0.0.0'
port = 5432
pooler_port = 20002
gtm_host = '127.0.0.1'
gtm_port = 6666
and coordinator like:
listen_addresses = '0.0.0.0'
port = 3456
pooler_port = 20003
gtm_host = '127.0.0.1'
gtm_port = 6666
others are default ,then i start datanode with :
/usr/postgres-xl-9.2/bin/postgres --datanode -D /data1/pgsql/data &
start coordinator with:
/usr/postgres-xl-9.2/bin/postgres --coordinator -D /data1/pgsql/coordinator &
when i connect to Postgres-XL use coordinator with :
psql -h 127.0.0.1 -p 3456 postgres;
when i want to create a database with:
create database test;
but failed,it shows :
No Datanode defined in cluster
You need to define at least 1 Datanode with CREATE NODE.
then i check the cluster with :
psql -h 127.0.0.1 -p 3456 postgres -c "select * from pgxc_node order by 1";
it shows just one node like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
-------------------+-----------+-----------+-----------+----------------+------------------+-----------
coordinator_node2 | C | 5432 | localhost | f | f | 738118815
and i connect to port 5432 it shows the similar information like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
------------+-----------+-----------+-----------+----------------+------------------+------------
data_node2 | C | 5432 | localhost | f | f | -923817565
what wrong with my Configuration?
Thanks all.

You need to define datanode something like this.
./psql -c "CREATE NODE datanode1 WITH (TYPE = 'datanode', PORT = 7777)" postgres -p 3456 -h 127.0.0.1

I'm answering this question a year after. However, I'd suggest you to use pgxc_ctl command line tool as suggested by the Postgres XL developers themselves. It will keep you away from the hectic tasks of manually configuring the cluster.
You can refer to the official documentation of Postgres XL for this. After you successfully installed pgxc_ctl command line tool, type in
prepare config minimal
in order to generate a basic configuration file to configure the cluster. Edit your changes in pgxc_ctl/pgxc_ctl.conf accordingly. Type in exit and finally you can initialize all the clusters if your configuration is okay. For this, type in
pgxc_ctl init all

Related

Run PostgreSQL streaming replication synchronous and asynchronous simultaneously

I am using PostgreSQL 14 and ubuntu as my OS.
I have done the database replication in synchronous mode
Now I want to add another server and the relationship between primary and new_standby node will be asynchronous.
Another problem, how to set application_name for different nodes? I have got same application_name=14/main for my two standby servers
Can anyone help me to solve these issues?
Yes, you can run it simultaneously.
You should have this configuration in your primary node:
listen_addresses = '*'
port = 5432
wal_level = hot_standby
max_wal_senders = 16
wal_keep_segments = 32
synchronous_commit = on
synchronous_standby_names = 'pgsql_0_node_0'
Restart the node to take the changes:
$ systemctl restart postgresql-14
Create the replication role:
$ CREATE ROLE replication_user WITH LOGIN PASSWORD 'PASSWORD' REPLICATION;
And configure this in your standby nodes:
Both:
wal_level = hot_standby
hot_standby = on
Sync:
standby_mode = 'on'
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_0 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Async:
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_1 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Restart the node to take the changes:
$ systemctl restart postgresql-14
Then, you can run this in your Primary node to see the replication nodes:
$ SELECT pid,usename,application_name,state,sync_state FROM pg_stat_replication;
pid | usename | application_name | state | sync_state
-------+------------------+------------------+-----------+------------
10951 | replication_user | pgsql_0_node_1 | streaming | async
10952 | replication_user | pgsql_0_node_0 | streaming | sync
(2 rows)

Can't connect PostGis to database and server

I followed these steps to set up QWC services https://github.com/qwc-services/qwc-services-core#quick-start and I can run the demo. But if load my own QGIS project, I receive the following error message:
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 WARNING ClearCapabilities[99]: Cached cleared : /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING PostGIS[99]: Connection to database failed
qwc-qgis-server_1 | could not connect to server: No such file or directory
qwc-qgis-server_1 | Is the server running locally and accepting
qwc-qgis-server_1 | connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 CRITICAL Server[99]: Error, Layer(s) measurement_b46e976f_2d0f_4bf0_942a_9d9462b40c3e not valid in project /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-config-service_1 | [2022-01-04 07:50:09,360] WARNING in config_generator: Skipping theme item '': Could not get capabilities for /ows/MeasurementDemo
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The generation of the configuration files resulted in a failure
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The configuration files were not updated!
qwc-config-service_1 | [2022-01-04 07:50:20,856] CRITICAL in config_generator: The generation of the permission files resulted in a failure.
qwc-config-service_1 | [2022-01-04 07:50:20,857] CRITICAL in config_generator: The permission files were not updated!
qwc-config-service_1 | [pid: 15|app: 0|req: 18/18] 172.18.0.11 () {30 vars in 408 bytes} [Tue Jan 4 07:50:05 2022] POST /generate_configs?tenant=default => generated 2881 bytes in 15083 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
As the error is quite similar to this question: PostgreSQL: Why psql can't connect to server?, I followed the answers but with no result.
ps -ef | grep postgres gives me the following result:
postgres 203911 1 0 07:35 ? 00:00:00 /usr/lib/postgresql/13/bin/postgres -D /var/lib/postgresql/13/main -c config_file=/etc/postgresql/13/main/postgresql.conf
Also I found the socket in
/var/run/postgresql/.s.PGSQL.5432
And I run the command
psql -h /var/run/postgresql/ GeoDB
But without result. After that I checked the ph_hba.conf File:
# "local" is for Unix domain socket connections only
local all all peer
Running the command pg_lsclusters gives me:
Ver Cluster Port Status Owner Data directory Log file
13 main 5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
Also after restarting the pg_ctlcluster and PostgreSQL the error remained the same.
Edit 1
After the answer from cnaimi I checked the postgresql.confFile:
# - Connection Settings -
#listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 100 # (change requires restart)
#superuser_reserved_connections = 3 # (change requires restart)
unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories
# (change requires restart)
#unix_socket_group = '*' # (change requires restart)
#unix_socket_permissions = 0777 # begin with 0 to use octal notation
# (change requires restart)
#bonjour = off # advertise server via Bonjour
# (change requires restart)
#bonjour_name = '' # defaults to the computer name
# (change requires restart)
But I can't find an error there as the port is 5432 and it listen to all adresses.
Edit 2
During my search I found several pg_service.conf Files:
./qwc-services/qwc-docker/wsgi-service/pg_service.conf
./qwc-services/qwc-docker/qgis-server/pg_service.conf
./qwc-services/qwc-docker/postgis/pg_service.conf
./qwc-services/qwc-docker/pg_service.conf
Each if them contain one or more credentials for databases like the one below:
[qwc_geodb]
host=qwc-postgis
port=5432
dbname=qwc_demo
user=qwc_service
password=qwc_service
sslmode=disable
The port is in all files correct, as far as I saw. But of course the db name and user/password are wrong. Does this could cause the error? Or does QWS get the credentials through the .qgs file?
Edit 3
Thanks to the hints from Devdatta Tengshe I set the host for PostgreSQL to 127.0.0.1. By using sudo docker-compose ps one can see the used container and their ports:
Name Command State Ports
-------------------------------------------------------------------------------------------------------------------------------
qwc-docker_qwc-admin-gui_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5031->9090/tcp
qwc-docker_qwc-api-gateway_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:8088->80/tcp,:::8088->80/tcp
qwc-docker_qwc-auth-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5017->9090/tcp
qwc-docker_qwc-config-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5010->9090/tcp
qwc-docker_qwc-data-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5012->9090/tcp
qwc-docker_qwc-elevation-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5002->9090/tcp
qwc-docker_qwc-fulltext-search-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5011->9090/tcp
qwc-docker_qwc-map-viewer_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5030->9090/tcp
qwc-docker_qwc-mapinfo-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5016->9090/tcp
qwc-docker_qwc-ogc-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5013->9090/tcp
qwc-docker_qwc-permalink-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5001->9090/tcp
qwc-docker_qwc-postgis_1 docker-entrypoint.sh postgres Up (healthy) 127.0.0.1:5439->5432/tcp
qwc-docker_qwc-qgis-server_1 /sbin/my_init Up 127.0.0.1:8001->80/tcp
qwc-docker_qwc-solr_1 docker-entrypoint.sh solr- ... Up 127.0.0.1:8983->8983/tcp
Can you check the postgres.conf file located in
/etc/postgresql/13/main/postgresql.conf
specially the parameter listen_address
Maybe you have to specify from which host you are listening.
But if the demo example is working the database configuration should be ok.
You can also check the port for postgres on postgres.conf and validate it's 5432.
There are a couple of things that need to be fixed to get this working.
I'm assuming that you have the Postgres Server running on the host machine, and not within any Docker container.
When you configured your QGIS Map file, you probably connected to localhost, and this information got saved in the .qgs file.
This is why your first error message says that it trying to connect to localhost, and no server was found. This error was thrown within the qwc docker container.
This error is occuring, because QGIS server (within the docker container) is not able to connect to the postgres server which is running on the host, using 'localhost' as the hostname
To solve this, you need to do the following:
In QGIS, connect to the Postgres Server using 127.0.0.1 and not localhost.
Save your qgs file using this new connection.
When you run the docker container for qwc, use --network="host" as the commandline parameter.
See: From inside of a Docker container, how do I connect to the localhost of the machine?
After this, the qgis server (within docker container) should be able to connect to the Postgres Server running on your host, using 127.0.0.1 as IP address.

Keycloak port 39008 and port scan

I did a port scan using nmap on my machine running Keycloak and was surprised to find that port 39008 was open. According to nmap:
39008/tcp open unknown
Using the following two commands I found that it is keycloak:
netstat -tulpn | grep 39008
tcp 0 0 0.0.0.0:39008 0.0.0.0:* LISTEN 17270/java
ps -Af | grep 17270
me 17270 17223 0 Dec22 ? 00:13:05 java ...-Djboss.home.dir=/.../keycloak-2.4.0.Final
I cannot find any reference to this port in the config or the docs.
What is this port used for?

pcp_attach_node gives EOFError in pgpool

I have successfully setup replication for my Postgres database using pg_pool.
Then I stopped the master server and checked the pool status. It is as like below
postgres=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role
---------+------------+------+--------+-----------+--------
0 | 10.140.0.9 | 5432 | 3 | 0.500000 | slave
1 | 10.140.0.7 | 5432 | 2 | 0.500000 | master
(2 rows)
Then I started the server, but it still shows the same status for the slave. So I used the following command to start the node:
/usr/sbin/pcp_node_info 10 10.140.0.9 5432 postgres postgres 1
But it is giving "EOFError" error. Please help to solve this issue.
Or please let me know a way to bring back the status 2 from status 3?
I solved the issue myself. In configuration the pcp port is 9898. Also there should be no space before password in pcp.conf file.
The pcp command should be as follows
/usr/sbin/pcp_node_info 10 localhost 9898 postgres postgres 1

How to install PostgreSQL 9.1 on OSX Lion

I downloaded PostgreSQL from the official website and ran the .dmg installer. After that I downloaded pgadmin3 and I am indeed able to connect to the database.
when I run 'psql' I get the following error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
After hours of googling I read about some $PATH issues, so I put this into my .bashrc:
export PATH=/Library/PostgreSQL/9.1/bin:$PATH
However, this doesn't solve the error above at all. After some more hours of googling I tried to run 'psql -l localhost -U postgres'. This gives another error:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (fe80::1) and accepting
TCP/IP connections on port 5432?
After some more googling I tried to edit /Library/PostgreSQL/9.1/data/pg_hba.conf and replaced all occurrences of 'md5' with 'trust'.
Then I changed user to postgres and executed 'pg_ctl stop' and 'pg_ctl start', switched back to my own user and tried to connect again, no luck.
Here are some more infos:
[~]$ which psql
/Library/PostgreSQL/9.1/bin/psql
ps aux | grep postgres
postgres 19022 0.0 0.0 2446096 484 ?? Ss 11:31PM 0:00.01 postgres: stats collector process
postgres 19021 0.0 0.0 2486532 1776 ?? Ss 11:31PM 0:00.01 postgres: autovacuum launcher process
postgres 19020 0.0 0.0 2486400 576 ?? Ss 11:31PM 0:00.03 postgres: wal writer process
postgres 19019 0.0 0.0 2486400 820 ?? Ss 11:31PM 0:00.05 postgres: writer process
postgres 19017 0.0 0.0 2446096 356 ?? Ss 11:31PM 0:00.01 postgres: logger process
postgres 19015 0.0 0.1 2486400 8216 s001 S 11:31PM 0:00.17 /Library/PostgreSQL/9.1/bin/postgres
sudo find / -name .s.PGSQL.5432
No file was found?!?
Update 1:
In /etc/sysctl.conf I added the values suggested by the installer's README:
kern.sysv.shmmax=1610612736
kern.sysv.shmall=393216
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
Before these settings, the installer quits with an error, afterwards the wizard appears and installs postgres (and again, using pgadmin3 works, so I assume that the database is running fine).
Run lsof on the master process to all of this. In your case it's 19015 (shown with my PID):
> sudo lsof -p 286 | awk '$5 == "unix" && $NF ~ /\// { print $NF }'
/tmp/.s.PGSQL.5432
You can leave off the awk, but basically it's getting the UNIX socket on which postgres is listening. From there, you can use the -h option to psql (but only include the directory).
> psql -h /tmp template1
template1=# \q
If that doesn't work, you can check the lsof output to show you what TCP port it's actually listening on, and if it's not 5432, use the -p option to postgres
If you are able to connect to the database through pgadmin3, then your database is running.
(1) I'd verify by running netstat -an | grep 5432 on the command line, as a paranoia check. You are in business i.e. you actually installed Postgresql if you get this below:
tcp4 0 0 *.5432 . LISTEN
tcp6 0 0 *.5432 . LISTEN
ffffff80133bfed8 stream 0 0 ffffff8013be8000 0 0 0 /tmp/.s.PGSQL.5432
(2) Speaking for myself, I found that just running psql gets me your psql error message. I had to run
psql -h 127.0.0.1 -p 5432 [-d database] -U postgres
to connect with the database. In other words, I had to explicitly specify IP and port. Since you were able to run psql and get a non-connect error message, the OS understands that psql is a command and you don't have a pathing issue.