Say we have 2 databases in postgresql: Database A(with dblink extension installed) and Database B (without dblink extension)
Copying table from B to A looks fairly straightforward:
select * into table_from_B from dblink('dbname=B', 'select id, name from some_table') as t(id bigint, name varchar(255));
What is the best way to copy/push a particular table from A to B using SQL?
Is it possible to do it using dblink without installing it on B ?
Related
I have many tables in different databases and want to bring them to a database.
It seems like I have to create foreign table in the database (where I want to merge them all) with schemas of all the tables.
I am sure, there is a way to automate this (by the way, I am going to use psql command) but I do not know where to start.
what I have found so far is I can use
select * from information_schema.columns
where table_schema = 'public' and table_name = 'mytable'
I added more detail explanation.
I wanted to copy tables from another database
the tables have same column names and data type
using postgres_fdw, I needed to set up a field name and data type for each tables (the table names are also same)
then, I want to union the tables have same name all to have one single table.
for that, I am going to add prefix on table
for instance, mytable in db1, mytable in db2, mytable in db3 as in
db1_mytable, db2_mytable, db3_mytable in my local database.
Thanks to Albe's comment, I managed it and now I need to figure out doing 4th step using psql command.
I have a set of applications accessing two different PostgreSQL 9.6 DBs on the same server. Due to some application limitations, one application accesses a handful of tables via FDW in one DB to the other.
Something like this:
DB1.fdw_table_a -> DB2.table_a
fdw_table_a is only used for inserts of log data. This table has an id column, which is a bigint sequence. The sequence exists in DB1 (on the foreign table) and in DB2 (the "real" table). This works as it should and all is well.
Now there's a need to have another application (again with limited access capabilities) perform inserts into the "real" table, DB2.table_a. In testing, I can see some inconsistencies in the id column, but no obvious issues have appeared.
I can see in the customer-facing environments that the DB1 FDW sequence is used as expected, but when inserts start directly on the DB2 'real' table, that sequence will start at 1 (as it has never been used).
Are there other things we should be considering in this environment?
Are there some issues that could arise from overlap in these two sequences inserting into the table?
The sequence only gets used if you omit the id column in the INSERT statement. But postgres_fdw will never omit a column, as you can see from the execution plan.
One way to solve the problem is to use a foreign table that does not contain the id column. Then any insert into that foreign table will use the sequence to populate that column.
The following plan from 2014 is still valid today.
=# CREATE SEQUENCE seq;
CREATE SEQUENCE
=# CREATE VIEW seq_view AS SELECT nextval('seq') as a;
CREATE VIEW
=# CREATE EXTENSION postgres_fdw;
CREATE EXTENSION
=# CREATE SERVER postgres_server
-# FOREIGN DATA WRAPPER postgres_fdw
-# OPTIONS (host 'localhost', port '5433', dbname 'postgres');
CREATE SERVER
=# CREATE USER MAPPING FOR PUBLIC SERVER postgres_server OPTIONS (password '');
CREATE USER MAPPING
=# CREATE FOREIGN TABLE foreign_seq_table (a bigint)
-# SERVER postgres_server OPTIONS (table_name 'seq_view');
CREATE FOREIGN TABLE
=# CREATE FUNCTION foreign_seq_nextval() RETURNS bigint AS
-# 'SELECT a FROM foreign_seq_table;' LANGUAGE SQL;
CREATE FUNCTION
=# CREATE TABLE tab (a int DEFAULT foreign_seq_nextval());
CREATE TABLE
=# INSERT INTO tab VALUES (DEFAULT), (DEFAULT), (DEFAULT);
INSERT 0 3
=# SELECT * FROM tab;
a
----
9
10
11
(3 rows)
https://paquier.xyz/postgresql-2/global-sequences-with-postgres_fdw-and-postgres-core/
I'm coming from a background in SQL Server where I would create temp tables using the:
select id
into #test
from table A
I've just moved into a PostGresql environment and I was hoping I could do the same, but I'm getting a syntax error. I did a search and it seems like you have to do a Create Table statement.
Is it not possible to easily create temp tables in Postgres?
Postgres supports SELECT INTO, so this should work fine:
SELECT id
INTO TEMP TABLE test
FROM a
You can also use CREATE TABLE AS:
CREATE TEMP TABLE test AS
SELECT id FROM a
This version is generally preferred, as the CREATE statement provides additional options, and can also be used in PL/pgSQL functions (where the SELECT INTO syntax has been hijacked for variable assignment).
I know that dblink can be queried directly like:
select * from dblink('kenyon_dblink','select * from test') as t1 (id integer,name varchar);
I wonder if I can query foreign servers without creating foreign tables, since my query strings are generated dynamically.
Yes you can, just open a dblink connection in the same session with dblink_connect() (named or unnamed).
Example:
Is there any shortcut for using dblink in Postgres?
Example with dynamic SQL:
Syntax error in function using dblink to replicate new data
This works for connections between PostgreSQL DBs, not for other RDBMS. Per documentation:
dblink is a module that supports connections to other PostgreSQL
databases.
I am using PostgreSQL 9.1. I need to transfer required columns from one table of one database into another table of another database, but not schema.
I found that dblink.sql file has to be there in share/contrib. But my contrib folder is empty. Where can I download the dblink.sql file and can execute my query?
When I execute the query now it shows an error message:
cross database reference is not possible ...
Can anyone help me how to transfer the data between two databases?
After you have installed the package into your system as detailed in the related question install the extension dblink into your database (the one you are running this code in, the foreign db does not need it):
CREATE EXTENSION dblink;
You can find code examples in the manual.
Here is a simple version of what I use to copy data between dbs:
First, create a FOREIGN SERVER
CREATE SERVER mydb
FOREIGN DATA WRAPPER postgresql
OPTIONS (hostaddr '111.111.111.111',port '5432',dbname 'mydb');
FOREIGN DATA WRAPPER postgresql was pre-installed in my case.
Then create function that opens a connection, removes old data (opotional), fetches new data, runs ANALYZE and closes the connection:
CREATE OR REPLACE FUNCTION f_tbl_sync()
RETURNS text AS
$BODY$
SELECT dblink_connect('mydb'); -- USER MAPPING for postgres, PW in .pgpass
TRUNCATE tbl; -- optional
INSERT INTO tbl
SELECT * FROM dblink(
'SELECT tbl_id, x, y
FROM tbl
ORDER BY tbl_id')
AS b(
tbl_id int
,x int
,y int)
ANALYZE tbl;
SELECT dblink_disconnect();
$BODY$
LANGUAGE sql VOLATILE;