Currently we have two dbs in AWS RDS (Postgres). One for Demo (Internal Testing) and another for beta environment. Both dbs have a schema called "strapi" which contains common data to display to user. Our use case is that whenever the admin updates the strapi data in demo db, it should reflect in beta as well. So I was searching for AWS services that supports our use case.
Searched for similar questions and I found people recommending AWS DMS. So went ahead and created endpoints and ran a full load task with "Truncate" option for Target Preparation Mode.
The task failed with following error
"RetCode: SQL_ERROR SqlState: 0A000 NativeError: 1 Message: ERROR: cannot truncate a table referenced in a foreign key constraint; Error while executing the query [1022502] (ar_odbc_stmt.c:4828). Failed to truncate table strapi.programs [1022502] (odbc_endpoint_imp.c:4372)"
Tried setting session_replication_role parameter to replica on target database in RDS and still getting the same error. Any help would be appreciated.
Note: I still am not sure whether AWS DMS is a good service for our use case.
Thanks in Advance.
I am still not sure whether AWS DMS is the correct service for my use case of creating a pipeline such that any changes in a particular schema in demo db should be reflected in beta db. But found the solution for the above DMS Task error.
"RetCode: SQL_ERROR SqlState: 0A000 NativeError: 1 Message: ERROR: cannot truncate a table referenced in a foreign key constraint; Error while executing the query [1022502] (ar_odbc_stmt.c:4828). Failed to truncate table strapi.programs [1022502] (odbc_endpoint_imp.c:4372)"
One can manually drop the foreign key constraints and once the full load migration task is completed, we can recreate the foreign key constraints
A simple script to drop foreign key constraints (Taken from https://dba.stackexchange.com/a/97047)
create table if not exists dropped_foreign_keys (
seq bigserial primary key,
sql text
);
do $$ declare t record;
begin
for t in select conrelid::regclass::varchar table_name, conname constraint_name,
pg_catalog.pg_get_constraintdef(r.oid, true) constraint_definition
from pg_catalog.pg_constraint r
where r.contype = 'f'
-- uncomment the below line for current schema only:
-- and r.connamespace = (select n.oid from pg_namespace n where n.nspname = current_schema())
loop
insert into dropped_foreign_keys (sql) values (
format('alter table %s add constraint %s %s',t.table_name, t.constraint_name, t.constraint_definition));
execute format('alter table %s drop constraint %s', t.table_name, t.constraint_name);
end loop;
end $$;
And to recreate the dropped foreign key constraints
do $$ declare t record;
begin
-- order by seq for easier troubleshooting when data does not satisfy FKs
for t in select * from dropped_foreign_keys order by seq loop
execute t.sql;
delete from dropped_foreign_keys where seq = t.seq;
end loop;
end $$;
Also note that for the full load task, there will not be any problem with migration as long as the constraints are created after the migration is completed on the target instance. But if you expect any updates on target instance for the tables in questions while the migration is running then you may see some issues with inconsistency because of dropping foreign key constraints.
Related
I would like to create trigger to execute function to truncate local database table and insert new data.
Trigger execution must start after new row have insert in foreign database table.
I have read a lot about creating triggers on foreign table, but for me its not working. Trigger seems to not execute function when new row will be inserted in foreign table. It seems like trigger cant see this new row insert event.
What I did:
Created foreign table in my local database, lets call it 'foreign_table'. I tested, I can read data.
Created function to truncate local table and insert new data:
CREATE or replace FUNCTION public.reset_insert_table()
RETURNS TRIGGER
LANGUAGE 'plpgsql'
SET search_path=public
AS $BODY$
BEGIN
create temporary table temporary_table_tmp
as select * from public.table1;
TRUNCATE TABLE public.table2;
insert into table2
select * from temporary_table_tmp;
DROP table temporary_table_tmp;
END;
$BODY$;
Created trigger to launch function 'reset_insert_table()'
CREATE TRIGGER local_table_update
AFTER INSERT
ON 'foreign_table'
FOR EACH ROW EXECUTE PROCEDURE reset_insert_table();
Made test: inserted new row in foreign database table 'foreign_table', but I cant see that table is truncated and new data is not inserted. Insertion to foreign_tale was done in foreign database.
Problem was also testing does this trigger function work, executing manually will produce error:
EXECUTE PROCEDURE reset_insert_table();
ERROR: syntax error at or near "execute"
Tried also CALL and SELECT.
I created same function for testing but instead defining 'RETURNS TRIGGER'used 'RETURNS VOID' and function is working.
Can anyone tell why my solution is not working and does trigger on foreign tables must see events happening in foreign tables?
According to your comments, you seem to be using logical replication.
While data modifications are replayed on the standby with logical replication, the parameter session_replication_role is set to replica to keep triggers and foreign key constraints from working.
If you want a trigger to be triggered by the replay of data via logical replication, you have to declare it as a replica trigger:
ALTER TABLE a2 ENABLE REPLICA TRIGGER trigger_name;
I have two databases: let's call them primary (which holds actual data) and fdw (which contains foreign-data-wrapper of data in primary db).
I create simple table in primary db:
create schema myschema;
create table myschema.foo (id bigint, whatever text);
create table myschema.foov as select * from foo;
I create foreign table in fdw db accessing primary table through view:
create extension postgres_fdw;
create server remote_docker foreign data wrapper postgres_fdw options (host 'primary', dbname 'postgres', port '5432');
create schema remote_myschema;
create user mapping for current_user server remote_docker options (user 'postgres');
create foreign table remote_myschema.foo (id bigint, whatever text) server remote_docker options (schema_name 'myschema', table_name 'foov');
When executing select * from remote_myschema.foo query, everything works ok.
The problem: if I didn't create view in primary db, the create foreign table command in fdw db passes without error anyway. I am able to discover the nonexistency of view in primary db only at time of query execution on fdw db.
The question: is somehow possible to detect that foreign table is bound to nonexistent original? I compared pg_class data of foreign table in both cases and didn't find any difference nor anything in documentation. The only way I know at this moment is catching exception
do $$
declare
ex boolean;
begin
begin
execute 'select null from remote_myschema.foo';
ex := true;
exception when others then
ex := false;
end;
raise notice '%', ex::text;
end;
$$;
which is awful.
Thanks!
Catching the exception is the only way. Unless views are in the habit of suddenly disappearing at your site, you don't have to test it every time you use the foreign table. Testing once, right after you created it, is good enough.
I wanted to remove all foreign key constraints from a schema. I was successful in dropping constraints from most of the tables but in few of them drop foreign key constraint query is getting stuck.
ALTER TABLE table_name DROP CONSTRAINT fkey_name;
I tried truncate cascade but it was also getting stuck. I deleted all rows from both the tables manually. Still getting stuck.
Edits: By getting stuck I mean query continues running for long time without any error message even though tables are empty.
Check for any dead locks using
SELECT * FROM pg_stat_activity;
If any then kill and run below sql,and then drop using
SELECT pg_terminate_backend(pid);
If Not solved check for any virtual transaction
SELECT database, gid FROM pg_prepared_xacts;
Rollback using
ROLLBACK PREPARED 'gid';
I have two databases db1 and db2.db2 has foreign table, say tableA whose schema has been imported from db1. A trigger runs on tableA in db1 for every row update.
Now using postgres_fdw I can fetch the records from db2 but unable to update any record on tableA due to that trigger function.Update works fine in case I disable the trigger.I need that trigger for audit log.
Please suggest me a suitable suggestion to resolve the issue.I am using postgres 9.6.
Make sure the user establishing the link has access to the audit tables.
You could also add the required schema to the trigger function search path:
CREATE OR REPLACE FUNCTION abc.mytrigger() RETURNS trigger AS
$BODY$BEGIN
[...] -- do something in the xyz schema
RETURN NEW;
END;$BODY$
LANGUAGE plpgsql
SET search_path = xyz;
I am using PostgreSQL 9.1. I need to transfer required columns from one table of one database into another table of another database, but not schema.
I found that dblink.sql file has to be there in share/contrib. But my contrib folder is empty. Where can I download the dblink.sql file and can execute my query?
When I execute the query now it shows an error message:
cross database reference is not possible ...
Can anyone help me how to transfer the data between two databases?
After you have installed the package into your system as detailed in the related question install the extension dblink into your database (the one you are running this code in, the foreign db does not need it):
CREATE EXTENSION dblink;
You can find code examples in the manual.
Here is a simple version of what I use to copy data between dbs:
First, create a FOREIGN SERVER
CREATE SERVER mydb
FOREIGN DATA WRAPPER postgresql
OPTIONS (hostaddr '111.111.111.111',port '5432',dbname 'mydb');
FOREIGN DATA WRAPPER postgresql was pre-installed in my case.
Then create function that opens a connection, removes old data (opotional), fetches new data, runs ANALYZE and closes the connection:
CREATE OR REPLACE FUNCTION f_tbl_sync()
RETURNS text AS
$BODY$
SELECT dblink_connect('mydb'); -- USER MAPPING for postgres, PW in .pgpass
TRUNCATE tbl; -- optional
INSERT INTO tbl
SELECT * FROM dblink(
'SELECT tbl_id, x, y
FROM tbl
ORDER BY tbl_id')
AS b(
tbl_id int
,x int
,y int)
ANALYZE tbl;
SELECT dblink_disconnect();
$BODY$
LANGUAGE sql VOLATILE;