Why can't I create pglogical subscription? - postgresql

I'm trying to setup replication from postgres DB source (pg 11.7, pglogical 2.2.1) to target (pg 13.5, pglogical 2.3.3)
The connectivity and access across DBs are configured and tested.
I've manually duplicated roles from source to target using pg_dump -g globals on source then psql -f globals.sql on target.
I've manually duplicated schema from source to target using pg_dump -Fc -s -f ~/schema.dmp mydatabase on source then pg_restore -d mydatabase schema.dmp on target.
I've modified each DBs postgres.conf with:
wal_level='logical'
max_worker_processes=10
max_replcation_slots=10
max_wal_senders=10
shared_preload_libraries='pglogical'
On both DBs I alter system set shared_preload_libraries = 'pglogical'; and restarted
On both DBs I CREATE EXTENSION pglogical;
On the source I create the node with SELECT pglogical.create_node(node_name := 'provider', dsn := 'host=<source_IP> port=5432 dbname=mydatabase user=pglogical password=<password>');
On the source I add all tables to replication set with SELECT pglogical.replication_set_add_all_tables('default', '{public}'::text[]);
On the source I add all sequences to replication set with SELECT pglogical.replication_set_add_all_sequences(set_name := 'default', schema_names := '{public}'::text[], synchronize_data := true );
On the target I create node with SELECT pglogical.create_node(node_name := 'subscriber', dsn := 'host=<target_IP> port=5432 dbname=mydatabase user=pglogical password=<password>');
Then finally I attempt to create subscription on source with `SELECT pglogical.create_subscription(subscription_name := 'subscription', provider_dsn := 'host=<source_IP> port=5432 dbname=mydatabase user=pglogical password=', replication_sets := '{default}'::text[]);
which results in the following error on the source:
ERROR: could not fetch remote node info: ERROR: function pglogical.pglogical_node_info() does not exist
LINE 1: ..., node_name, sysid, dbname, replication_sets FROM pglogical....
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
and this error in found in journal on the target:
LOG: connection authorized: user=pglogical database=mydatabase SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off)
ERROR: function pglogical.pglogical_node_info() does not exist at character 65
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
STATEMENT: SELECT node_id, node_name, sysid, dbname, replication_sets FROM pglogical.pglogical_node_info()
I've tried that pglogical.create_subscription function with all typecasts included, doesn't change anything.
On the target I see:
postgres=# select * from pglogical.pglogical_node_info()
;
node_id | node_name | sysid | dbname | replication_sets
------------+------------+---------------------+----------+----------------------------------------------
2941155235 | subscriber | 7067748448099432568 | postgres | "걵wN`PXU","\x04\x0B鐣wNPXU","\x0FNl7wNxPXU"
(1 row)
on the source I see:
postgres=# select * from pglogical.pglogical_node_info()
;
node_id | node_name | sysid | dbname | replication_sets
------------+-----------------------+---------------------+----------+-------------------------------------------
2678724765 | provider | 6825764350976429997 | postgres | "\x08P\x180U"," \x03;%\x180U","BBԝ\x180U"
(1 row)
Why can't I create this subscription? Is streaming from PG11.7 to PG13.5 supported by pglogical?

I think your create_subscription statement is incorrect.
pglogical.create_subscription(subscription_name := 'subscription', provider_dsn := 'host=<target_IP> port=5432 dbname=mydatabase user=pglogical password=', replication_sets := '{default}'::text[]);
You are including <target_IP> instead of <source_IP>
Another possible issue - I'm not sure about your syntax for replication sets. You may want to remove that. You don't need to list them again here as you've already set it up at the node level.
Further resources:
pglogical docs. Search for references to create_subscription
Helpful AWS blog post walkthrough for using pglogical. I am using now myself to upgrade PG.

After creating another much simpler test bed, I've discovered what was likely the issue here. The pglogical subscriptions are not managed globally. They are specific to each database. For example, to create node and subscription one must explicitly connect to the database to replicated first.
postgres=# SELECT subscription_name, status FROM pglogical.show_subscription_status();
subscription_name | status
-------------------+--------
(0 rows)
postgres=# \c mydatabase
You are now connected to database "mydatabase" as user "postgres".
mydatabase=# SELECT subscription_name, status FROM pglogical.show_subscription_status();
subscription_name | status
-------------------+-------------
subscription | replicating
(1 row)

Related

Search is not working with lowercase LIKE for russian characters

What is the reason for this query give no results:
SELECT name FROM users WHERE LOWER(name) LIKE LOWER('%после%');
When this works fine:
SELECT name FROM users WHERE LOWER(name) LIKE LOWER('%После%');
Name is 'Последователь'. If i use name like 'Post', then search works fine.
Version: PostgreSQL 11.2 (Ubuntu 11.2-100) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0, 64-bit
Server and db encoding is UTF8. Client encoding is UNICODE.
The lower function works according to the database collation, so the result will vary depending on how your database is defined.
It will work well in this case:
test=# CREATE DATABASE rus TEMPLATE template0
ENCODING UTF8 LC_COLLATE "ru_RU.utf8" LC_CTYPE "ru_RU.utf8";
CREATE DATABASE
test=# \c rus
You are now connected to database "rus" as user "postgres".
rus=# SELECT LOWER('%после%') = LOWER('%После%');
?column?
----------
t
(1 row)
But it won't work with the C collation, because that does not know how to properly lower case cyrillic characters:
rus=# \c test
You are now connected to database "test" as user "postgres".
test=# DROP DATABASE rus;
DROP DATABASE
test=# CREATE DATABASE rus TEMPLATE template0
ENCODING UTF8 LC_COLLATE "C" LC_CTYPE "C";
CREATE DATABASE
test=# \c rus
You are now connected to database "rus" as user "postgres".
rus=# SELECT LOWER('%после%') = LOWER('%После%');
?column?
----------
f
(1 row)
So if you want your query to work well, make sure that you are using a collation that knows how to convert the characters to upper and lower case.
Functions LOWER and UPPER do not work correct if database has wrong locale.
Step-1. View locale of databases in Postgres:
\l
Step-2. View available locales in terminal:
locale -a
or by your language, for example ru:
locale -a | grep ru
Step-3. Use locale name from Step-2(for example ru_RU.utf8) in updating database YOUR_DATABASE_NAME in Postgres:
update pg_database set datcollate='ru_RU.utf8', datctype='ru_RU.utf8' where datname='YOUR_DATABASE_NAME';
or you can create new database with locale:
CREATE DATABASE NEW_DATABASE_NAME TEMPLATE template0 ENCODING UTF8 LC_COLLATE "ru_RU.UTF-8" LC_CTYPE "ru_RU.UTF-8";
PS My macOS has name ru_RU.UTF-8, Ubuntu has name ru_RU.utf8
Look Closer. LIKE statement is sometimes case sensitive. Depens on the program you use
The first statement is :
SELECT name FROM users WHERE LOWER(name) LIKE LOWER('%после%');
while the second is :
SELECT name FROM users WHERE LOWER(name) LIKE LOWER('%После%');
What you seek is
П оследователь
note that in the first statement you wrote with (п) while the second one is (П)

AWS RDS PostgreSQL 9.5.4 Extension postgis_tiger_geocoder Missing Soundex?

I am attempting to install the AWS "Approved" PostgreSql Extension on our on large RDS instance but every time I at the point I attempt to 'create extension postgis_tiger_geocoder' I get this:
SQL Error [42883]: ERROR: function soundex(character varying) does not exist
I have spent a good bit of time reading the AWS / postgis / postgresql forums but unfortunately haven't found the writing on the wall.
Steps Taken
Installed the POSTGIS extension
create EXTENSION postgis;
Installed the FuzzyStrMatch Extension which contains the soundex function (verified)
create EXTENSION fuzzystrmatch;
Finally when I run this create extension I get the error above
create extension postgis_tiger_geocoder;
SQL Error [42883]: ERROR: function soundex(character varying) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 57558
org.postgresql.util.PSQLException: ERROR: function soundex(character varying) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 57558
Things I have tried:
set search_path = <schema_name>, public
Followed here:
Installing PostgreSQL Extension to all schemas
Dug deeply into postgis installation documentation
Read through RDS documentation on adding Extensions...
If anyone has had to deal with this frustration on AWS I will happily swap a few of the remaining hairs left on my head as I have not been able to work around this.
Results of \dx+
Objects in extension "fuzzystrmatch"
Object Description
--------------------------------------------------------------------------------
function <schema>.difference(...)
function <schema>.dmetaphone_alt(...)
function <schema>.dmetaphone(...)
function <schema>.levenshtein_less_equal(...)
function <schema>.levenshtein_less_equal(...)
function <schema>.levenshtein(...)
function <schema>.levenshtein(...)
function <schema>.metaphone(...)
function <schema>.soundex(...)
function <schema>.text_soundex(...)
(10 rows)
Results of \dfS+ soundex
List of functions
Schema | Name | Result data type | Argument data types | Type | Volatility | Owner | Security | Access privileges | Language | Source code | Description
--------+------+------------------+---------------------+------+------------+-------+----------+-------------------+----------+-------------+-------------
(0 rows)
Had the same problem, resolved it by altering search_path for database and reconnect before creating extension postgis_tiger_geocoder. Look for the FIX part :
-- Postgis Installation
------------------------------------------------------------------------------------------------------------------------------------------------
-- PostGIS AWS Configuration --
-- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.PostGIS --
------------------------------------------------------------------------------------------------------------------------------------------------
-- On postgis schema
SET SCHEMA '${POSTGIS_SCHEMA_NAME}';
-- Step 2: Load the PostGIS Extensions
create extension postgis;
create extension fuzzystrmatch;
-- FIX : To avoid "ERROR: function soundex(character varying) does not exist", change schema and reconnect
ALTER DATABASE ${DATABASE_NAME} SET search_path=${POSTGIS_SCHEMA_NAME};
\connect ${DATABASE_NAME};
-- End FIX
create extension postgis_tiger_geocoder;
create extension postgis_topology;
-- Step 3: Transfer Ownership of the Extensions to the rds_superuser Role
alter schema tiger owner to ${MASTER_USER};
alter schema tiger_data owner to ${MASTER_USER};
alter schema topology owner to ${MASTER_USER};
-- Step 4: Transfer Ownership of the Objects to the rds_superuser Role
CREATE FUNCTION exec(text) returns text language plpgsql volatile AS $f$ BEGIN EXECUTE $1; RETURN $1; END; $f$;
SELECT exec('ALTER TABLE ' || quote_ident(s.nspname) || '.' || quote_ident(s.relname) || ' OWNER TO ${MASTER_USER};')
FROM (
SELECT nspname, relname
FROM pg_class c JOIN pg_namespace n ON (c.relnamespace = n.oid)
WHERE nspname in ('tiger','topology') AND
relkind IN ('r','S','v') ORDER BY relkind = 'S')
s;
-- Adding postgis to default schema
ALTER DATABASE ${DATABASE_NAME} SET search_path=${SCHEMA_NAME},${POSTGIS_SCHEMA_NAME};
Had the same problem. Turns out that all the fuzzystrmatch's functions were created inside the wrong schema.
Connected with psql command line, I used the drop extension command to restart the process of creating the extensions:
drop extension postgis_topology;
drop extension postgis;
drop extension fuzzystrmatch;
Then, just to be sure, disconnected using \q.
Connected psql again.
Set the schema to public:
set schema 'public';
Then, follow the process described in AWS RDS Docs
create extension postgis;
create extension fuzzystrmatch;
create extension postgis_tiger_geocoder;
create extension postgis_topology;

pg_stat_statements enabled, but the table does not exist

I've postgresql-9.4 up and running, and I've enabled pg_stat_statements module lately by the help of official documentation.
But I'm getting following error upon usage:
postgres=# SELECT * FROM pg_stat_statements;
ERROR: relation "pg_stat_statements" does not exist
LINE 1: SELECT * FROM pg_stat_statements;
postgres=# SELECT pg_stat_statements_reset();
ERROR: function pg_stat_statements_reset() does not exist
LINE 1: SELECT pg_stat_statements_reset();
I'm logged in to psql with the postgres user.
I've also checked the available extension lists:
postgres=# SELECT * FROM pg_available_extensions WHERE name = 'pg_stat_statements'
;
name | default_version | installed_version | comment
--------------------+-----------------+-------------------+-----------------------------------------------------------
pg_stat_statements | 1.2 | | track execution statistics of all SQL statements executed
(1 row)
And here's the results of the extension versions query:
postgres=# SELECT * FROM pg_available_extension_versions WHERE name = 'pg_stat_statements';
name | version | installed | superuser | relocatable | schema | requires | comment
--------------------+---------+-----------+-----------+-------------+--------+----------+-----------------------------------------------------------
pg_stat_statements | 1.2 | f | t | t | | | track execution statistics of all SQL statements executed
(1 row)
Any help will be appreciated.
Extension isn't installed:
SELECT *
FROM pg_available_extensions
WHERE
name = 'pg_stat_statements' and
installed_version is not null;
If the table is empty, create the extension:
CREATE EXTENSION pg_stat_statements;
I've faced with this issue at configuring Percona Monitoring and Management (PMM) because by some strange reason PMM connecting to database with name postgres, so pg_stat_statements extension have to be created in this database:
yourdb# \c postgres
postgres# CREATE EXTENSION pg_stat_statements SCHEMA public;
Follow below steps:
Create the extension
CREATE EXTENSION pg_stat_statements;
Change in config
alter system set shared_preload_libraries='pg_stat_statements';
Restart
$ systemctl restart postgresql
Verify changes applied or not.
select * from pg_file_Settings where name='shared_preload_libraries';
The applied attribute must be 'true'.
I Had the same issue when deploying the environment using liquibase for the first time.
I understand that my reply maybe is not related with your problem but was the first google result so I think that other guys like me can arrive here with my the same Liquibase Issue.
These are PosGreSQL metadata tables that are retrieved by liquibase when you generate your first xml file.
In my case it only was useless autogenerated code, so I solved it deleteing these lines:
<changeSet author="martinlarizzate (generated)" id="1588181532394-7">
<createView fullDefinition="false" viewName="pg_stat_statements"> SELECT pg_stat_statements.userid,
pg_stat_statements.dbid,
pg_stat_statements.queryid,
pg_stat_statements.query,
pg_stat_statements.calls,
pg_stat_statements.total_time,
pg_stat_statements.min_time,
pg_stat_statements.max_time,
pg_stat_statements.mean_time,
pg_stat_statements.stddev_time,
pg_stat_statements.rows,
pg_stat_statements.shared_blks_hit,
pg_stat_statements.shared_blks_read,
pg_stat_statements.shared_blks_dirtied,
pg_stat_statements.shared_blks_written,
pg_stat_statements.local_blks_hit,
pg_stat_statements.local_blks_read,
pg_stat_statements.local_blks_dirtied,
pg_stat_statements.local_blks_written,
pg_stat_statements.temp_blks_read,
pg_stat_statements.temp_blks_written,
pg_stat_statements.blk_read_time,
pg_stat_statements.blk_write_time
FROM pg_stat_statements(true) pg_stat_statements(userid, dbid, queryid, query, calls, total_time, min_time, max_time, mean_time, stddev_time, rows, shared_blks_hit, shared_blks_read, shared_blks_dirtied, shared_blks_written, local_blks_hit, local_blks_read, local_blks_dirtied, local_blks_written, temp_blks_read, temp_blks_written, blk_read_time, blk_write_time);</createView>
</changeSet>

How to delete replication slot in postgres 9.4

I have replication slot which I want to delete but when I do delete I got an error that I can't delete from view. Any ideas?
postgres=# SELECT * FROM pg_replication_slots ;
slot_name | plugin | slot_type | datoid | database | active | xmin | catalog_xmin | restart_lsn
--------------+--------------+-----------+--------+----------+--------+------+--------------+-------------
bottledwater | bottledwater | logical | 12141 | postgres | t | | 374036 | E/FE8D9010
(1 row)
postgres=# delete from pg_replication_slots;
ERROR: cannot delete from view "pg_replication_slots"
DETAIL: Views that do not select from a single table or view are not automatically updatable.
HINT: To enable deleting from the view, provide an INSTEAD OF DELETE trigger or an unconditional ON DELETE DO INSTEAD rule.
postgres=#
Use pg_drop_replication_slot:
select pg_drop_replication_slot('bottledwater');
See the docs and this blog.
The replication slot must be inactive, i.e. no active connections. So if there's a streaming replica using the slot you must stop the streaming replica. Or you can change its recovery.conf so it doesn't use a slot anymore and restart it.
As a complement to the accepted answer, I'd like to mention that following command will not fail in case the slot does not exist (this was useful for me because I scripted that).
select pg_drop_replication_slot(slot_name) from pg_replication_slots where slot_name = 'bottledwater';

Extension exists but uuid_generate_v4 fails

At amazon ec2 RDS Postgresql:
=> SHOW rds.extensions;
rds.extensions
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
btree_gin,btree_gist,chkpass,citext,cube,dblink,dict_int,dict_xsyn,earthdistance,fuzzystrmatch,hstore,intagg,intarray,isn,ltree,pgcrypto,pgrowlocks,pg_trgm,plperl,plpgsql,pltcl,postgis,postgis_tiger_geocoder,postgis_topology,sslinfo,tablefunc,tsearch2,unaccent,uuid-ossp
(1 row)
As you can see, uuid-ossp extension does exist. However, when I'm calling the function for generation uuid_v4, it fails:
CREATE TABLE my_table (
id uuid DEFAULT uuid_generate_v4() NOT NULL,
name character varying(32) NOT NULL,
);
What's wrong with this?
The extension is available but not installed in this database.
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
If the extension is already there but you don't see the uuid_generate_v4() function when you do a describe functions \df command then all you need to do is drop the extension and re-add it so that the functions are also added. Here is the issue replication:
db=# \df
List of functions
Schema | Name | Result data type | Argument data types | Type
--------+------+------------------+---------------------+------
(0 rows)
CREATE EXTENSION "uuid-ossp";
ERROR: extension "uuid-ossp" already exists
DROP EXTENSION "uuid-ossp";
CREATE EXTENSION "uuid-ossp";
db=# \df
List of functions
Schema | Name | Result data type | Argument data types | Type
--------+--------------------+------------------+---------------------------+--------
public | uuid_generate_v1 | uuid | | normal
public | uuid_generate_v1mc | uuid | | normal
public | uuid_generate_v3 | uuid | namespace uuid, name text | normal
public | uuid_generate_v4 | uuid | | normal
db=# select uuid_generate_v4();
uuid_generate_v4
--------------------------------------
b19d597c-8f54-41ba-ba73-02299c1adf92
(1 row)
What probably happened is that the extension was originally added to the cluster at some point in the past and then you probably created a new database within that cluster afterward. If that was the case then the new database will only be "aware" of the extension but it will not have the uuid functions added which happens when you add the extension. Therefore you must re-add it.
Looks like the extension is not installed in the particular database you require it.
You should connect to this particular database with
\CONNECT my_database
Then install the extension in this database
CREATE EXTENSION "uuid-ossp";
Step #1: re-install uuid-ossp extention into the exact schema:
If this is a fresh installation you can skip SET and DROP. Credits to #atomCode (details)
SET search_path TO public;
DROP EXTENSION IF EXISTS "uuid-ossp";
CREATE EXTENSION "uuid-ossp" SCHEMA public;
After this, you should see uuid_generate_v4() function IN THE RIGHT SCHEMA (when execute \df query in psql command-line prompt).
Step #2: use fully-qualified names (with schemaname. qualifier):
For example:
CREATE TABLE public.my_table (
id uuid DEFAULT public.uuid_generate_v4() NOT NULL,
If you've changed the search_path, specify the public schema in the function call:
public.uuid_generate_v4()
This worked for me.
create extension IF NOT EXISTS "uuid-ossp" schema pg_catalog version "1.1";
make sure the extension should by on pg_catalog and not in your schema...
Just add this code to the Beginning of your script
DROP EXTENSION IF EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
Maybe It was the same I was facing. The uuid_generate_v4 was from the public schema and I was trying to run it in a specific schema, so to fix it I did:
SET search_path TO specific_schema;
INSERTO INTO my_table VALUES public.uuid_generate_v4();
You can check the schema where your function is running:
\df uuid_generate_v4
Or
SELECT n.nspname, p.probin, p.proname
FROM
pg_proc p
LEFT JOIN pg_namespace n ON p.pronamespace = n.oid
WHERE p.proname like 'uuid_generate_v4';
You can check info related to the extension of the uuid-ossp like this:
SELECT * FROM pg_extension WHERE extname LIKE 'uuid-ossp';
You can add this extension case you don't have it already:
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
if you do it from unix command (apart from PGAdmin) dont forget to pass the DB as a parameter. otherwise this extension will not be enabled when executing requests on this DB
psql -d -c "create EXTENSION pgcrypto;"
in my case were 3 steps. Create the database, connect to the database and create the extension. The important step is the second one, "connect to the database", and you can notice the line without ";" cause is a command and not a SQL sentence.
CREATE DATABASE database_name_here;
\connect database_name_here
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";