I have a set of applications accessing two different PostgreSQL 9.6 DBs on the same server. Due to some application limitations, one application accesses a handful of tables via FDW in one DB to the other.
Something like this:
DB1.fdw_table_a -> DB2.table_a
fdw_table_a is only used for inserts of log data. This table has an id column, which is a bigint sequence. The sequence exists in DB1 (on the foreign table) and in DB2 (the "real" table). This works as it should and all is well.
Now there's a need to have another application (again with limited access capabilities) perform inserts into the "real" table, DB2.table_a. In testing, I can see some inconsistencies in the id column, but no obvious issues have appeared.
I can see in the customer-facing environments that the DB1 FDW sequence is used as expected, but when inserts start directly on the DB2 'real' table, that sequence will start at 1 (as it has never been used).
Are there other things we should be considering in this environment?
Are there some issues that could arise from overlap in these two sequences inserting into the table?
The sequence only gets used if you omit the id column in the INSERT statement. But postgres_fdw will never omit a column, as you can see from the execution plan.
One way to solve the problem is to use a foreign table that does not contain the id column. Then any insert into that foreign table will use the sequence to populate that column.
The following plan from 2014 is still valid today.
=# CREATE SEQUENCE seq;
CREATE SEQUENCE
=# CREATE VIEW seq_view AS SELECT nextval('seq') as a;
CREATE VIEW
=# CREATE EXTENSION postgres_fdw;
CREATE EXTENSION
=# CREATE SERVER postgres_server
-# FOREIGN DATA WRAPPER postgres_fdw
-# OPTIONS (host 'localhost', port '5433', dbname 'postgres');
CREATE SERVER
=# CREATE USER MAPPING FOR PUBLIC SERVER postgres_server OPTIONS (password '');
CREATE USER MAPPING
=# CREATE FOREIGN TABLE foreign_seq_table (a bigint)
-# SERVER postgres_server OPTIONS (table_name 'seq_view');
CREATE FOREIGN TABLE
=# CREATE FUNCTION foreign_seq_nextval() RETURNS bigint AS
-# 'SELECT a FROM foreign_seq_table;' LANGUAGE SQL;
CREATE FUNCTION
=# CREATE TABLE tab (a int DEFAULT foreign_seq_nextval());
CREATE TABLE
=# INSERT INTO tab VALUES (DEFAULT), (DEFAULT), (DEFAULT);
INSERT 0 3
=# SELECT * FROM tab;
a
----
9
10
11
(3 rows)
https://paquier.xyz/postgresql-2/global-sequences-with-postgres_fdw-and-postgres-core/
Related
I have some tables in PostgreSQL 12.9 that were declared as something like
-- This table is written in old style
create table old_style_table_1 (
id bigserial not null primary key,
...
);
-- This table uses new feature
create table new_style_table_2 (
id bigint generated by default as identity,
...
);
Second table seems to be declared using the identity flag introduced in 10th version.
Time went by, and we have partitioned the old tables, while keeping the original sequences:
CREATE TABLE partitioned_old_style_table_1 (LIKE old_style_table_1 INCLUDING DEFAULTS) PARTITION BY HASH (user_id);
CREATE TABLE partitioned_new_style_table_2 (LIKE new_style_table_2 INCLUDING DEFAULTS) PARTITION BY HASH (user_id);
DDL for their id columns seems to be id bigint default nextval('old_style_table_1_id_seq') not null and id bigint default nextval('new_style_table_2_id_seq') not null.
Everything has worked fine so far. Partitioned tables proved to be a great boon and we decided to retire the old tables by dropping them.
DROP TABLE old_style_table_1, new_style_table_2;
-- [2BP01] ERROR: cannot drop desired object(s) because other objects depend on them
-- Detail: default value for column id of table old_style_table_1 depends on sequence old_style_table_1_id_seq
-- default value for column id of table new_style_table_2 depends on sequence new_style_table_2_id_seq
After some pondering I've found out that sequences may have owners in postgres, so I opted to change them:
ALTER SEQUENCE old_style_table_1_id_seq OWNED BY partitioned_old_style_table_1.id;
DROP TABLE old_style_table_1;
-- Worked out flawlessly
ALTER SEQUENCE new_style_table_2_id_seq OWNED BY partitioned_new_style_table_2.id;
ALTER SEQUENCE new_style_table_2_id_seq OWNED BY NONE;
-- Here's the culprit of the question:
-- [0A000] ERROR: cannot change ownership of identity sequence
So, apparently the fact that this column has pg_attribute.attidentity set to 'd' forbids me from:
• changing the default value of the column:
ALTER TABLE new_style_table_2 ALTER COLUMN id SET DEFAULT 0;
-- [42601] ERROR: column "id" of relation "new_style_table_2" is an identity column
• dropping the default value:
ALTER TABLE new_style_table_2 ALTER COLUMN id DROP DEFAULT;
-- [42601] ERROR: column "id" of relation "new_style_table_2" is an identity column
-- Hint: Use ALTER TABLE ... ALTER COLUMN ... DROP IDENTITY instead.
• dropping the identity, column or the table altogether (new tables already depend on the sequence):
ALTER TABLE new_style_table_2 ALTER COLUMN id DROP IDENTITY IF EXISTS;
-- or
ALTER TABLE new_style_table_2 DROP COLUMN id;
-- or
DROP TABLE new_style_table_2;
-- result in
-- [2BP01] ERROR: cannot drop desired object(s) because other objects depend on them
-- default value for column id of table partitioned_new_style_table_2 depends on sequence new_style_table_2_id_seq
I've looked up the documentation, it provides the way to SET IDENTITY or ADD IDENTITY, but no way to remove it or to change to a throwaway sequence without attempting to drop the existing one.
➥ So, how am I able to remove an identity flag from the column-sequence pair so it won't affect other tables that use this sequence?
UPD: Tried running UPDATE pg_attribute SET attidentity='' WHERE attrelid=16816; on localhost, still receive [2BP01] and [0A000]. :/
Though I managed to execute the DROP DEFAULT value bit, but it seems like a dead end.
I don't think there is a safe and supported way to do that (without catalog modifications). Fortunately, there is nothing special about sequences that would make dropping them a problem. So take a short down time and:
remove the default value that uses the identity sequence
record the current value of the sequence
drop the table
create a new sequence with an appropriate START value
use the new sequence to set new default values
If you want an identity column, you should define it on the partitioned table, not on one of the partitions.
I have two databases: let's call them primary (which holds actual data) and fdw (which contains foreign-data-wrapper of data in primary db).
I create simple table in primary db:
create schema myschema;
create table myschema.foo (id bigint, whatever text);
create table myschema.foov as select * from foo;
I create foreign table in fdw db accessing primary table through view:
create extension postgres_fdw;
create server remote_docker foreign data wrapper postgres_fdw options (host 'primary', dbname 'postgres', port '5432');
create schema remote_myschema;
create user mapping for current_user server remote_docker options (user 'postgres');
create foreign table remote_myschema.foo (id bigint, whatever text) server remote_docker options (schema_name 'myschema', table_name 'foov');
When executing select * from remote_myschema.foo query, everything works ok.
The problem: if I didn't create view in primary db, the create foreign table command in fdw db passes without error anyway. I am able to discover the nonexistency of view in primary db only at time of query execution on fdw db.
The question: is somehow possible to detect that foreign table is bound to nonexistent original? I compared pg_class data of foreign table in both cases and didn't find any difference nor anything in documentation. The only way I know at this moment is catching exception
do $$
declare
ex boolean;
begin
begin
execute 'select null from remote_myschema.foo';
ex := true;
exception when others then
ex := false;
end;
raise notice '%', ex::text;
end;
$$;
which is awful.
Thanks!
Catching the exception is the only way. Unless views are in the habit of suddenly disappearing at your site, you don't have to test it every time you use the foreign table. Testing once, right after you created it, is good enough.
I am having trouble thinking of a way to copy three fields out of a database into and append them to another table along with the current date. Basically what I want to do is:
DB-A: ID (N9), Name (C69), Phone (N15) {and a list of other fields I dont care about}
DB-B: Date (Todays date/time), Nane, Address, Phone (as above)
Would be great is this was a trigger in the DB on add or update of DB-A.
Greg
Quick and dirty using postgres_fdw
CREATE EXTENSION IF NOT EXISTS postgres_fdw ;
CREATE SERVER extern_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'foreignserver.co.uk', port '5432', dbname 'mydb');
CREATE USER MAPPING FOR myuser SERVER extern_server OPTIONS (user 'anotheruser');
-- Creating a foreign table based on table t1 at the server described above
CREATE FOREIGN TABLE foreign_t1 (
dba INT,
name VARCHAR(9),
phone VARCHAR(15)
)
SERVER extern_server OPTIONS (schema_name 'public', table_name 't1');
--Inserting data to a new table + date
INSERT INTO t2 SELECT dba,name,phone,CURRENT_DATE FROM foreign_t1;
-- Or just retrieving what you need placing the current date as a column
SELECT dba,name,phone,CURRENT_DATE FROM foreign_t1;
Here i need to select a column name by using function(stored procedure) which is present in other database table using PostgreSQL.
I have sql server query as shown below.
Example:
create procedure sp_testing
as
if not exists ( select ssn from testdb..testtable) /*ssn is the column-name of testtable which exists in testdb database */
...
Q: Can i do the same in PostgreSQL?
Your question is not very clear, but if you want to know if a column by a certain name exists in a table by a certain name in a remote PostgreSQL database, then you should first set up a foreign data wrapper, which is a multi-stage process. Then to test the existence of a certain column in a table you need to formulate a query that conforms to the standards of the particular DBMS that you are connecting to. Use the remote information_schema.tables table for optimal compatibility (which is here specified as remote_tables which you must have defined with a prior CREATE FOREIGN TABLE command):
CREATE FUNCTION sp_testing () AS $$
BEGIN
PERFORM *
FROM remote_tables
WHERE table_name = 'testtable'
AND column_name = 'ssn';
IF NOT FOUND THEN
...
END IF;
END;
$$ LANGUAGE plpgsql;
If you want to connect to another type of DBMS, you need to write some custom function in f.i. C or perl and then call that from within a PostgreSQL function on your local machine. The test on the column is then best done inside the function which should therefore take connection parameters, table name and column name as parameters, and return a boolean to inform the result.
Before you start testing this, make sure that you read all the documentation on connecting to remote servers and learning PL/pgSQL first would also be a nice gesture to demonstrate your own efforts before you ask for help.
I am using PostgreSQL 9.1. I need to transfer required columns from one table of one database into another table of another database, but not schema.
I found that dblink.sql file has to be there in share/contrib. But my contrib folder is empty. Where can I download the dblink.sql file and can execute my query?
When I execute the query now it shows an error message:
cross database reference is not possible ...
Can anyone help me how to transfer the data between two databases?
After you have installed the package into your system as detailed in the related question install the extension dblink into your database (the one you are running this code in, the foreign db does not need it):
CREATE EXTENSION dblink;
You can find code examples in the manual.
Here is a simple version of what I use to copy data between dbs:
First, create a FOREIGN SERVER
CREATE SERVER mydb
FOREIGN DATA WRAPPER postgresql
OPTIONS (hostaddr '111.111.111.111',port '5432',dbname 'mydb');
FOREIGN DATA WRAPPER postgresql was pre-installed in my case.
Then create function that opens a connection, removes old data (opotional), fetches new data, runs ANALYZE and closes the connection:
CREATE OR REPLACE FUNCTION f_tbl_sync()
RETURNS text AS
$BODY$
SELECT dblink_connect('mydb'); -- USER MAPPING for postgres, PW in .pgpass
TRUNCATE tbl; -- optional
INSERT INTO tbl
SELECT * FROM dblink(
'SELECT tbl_id, x, y
FROM tbl
ORDER BY tbl_id')
AS b(
tbl_id int
,x int
,y int)
ANALYZE tbl;
SELECT dblink_disconnect();
$BODY$
LANGUAGE sql VOLATILE;