I have master-slave architecture. On the slave I have an Oracle database with two schemas, e.g. TEST1 and TEST2.
I have all objects (e.g. EMPLOYEES) stored in schema TEST1, and user TEST1 (or admin) has given read only privileges on TEST1.EMPLOYEES to TEST2, so when I use TEST1.EMPLOYEES in a query on the Oracle database I can access its data.
How can I implement the same using Oracle foreign data wrapper in postgres 9.5 because I have credentials for TEST2 and not TEST1?
When I try to access the foreign table it give an error saying that TEST2.EMPLOYEES does not exist.
You can easily do that if you define the user mapping with the credentials of user TEST2 and the foreign table with the schema option, i.e.
CREATE FOREIGN TABLE ... OPTIONS (schema 'TEST1', table 'EMPLOYEES');
Related
For a long time I have been working only with Oracle Databases and I haven't had much contact with PostgreSQL.
So now, I have a few questions for people who are closer to Postgres.
Is it possible to create a connection from Postgres to Oracle (oracle_fdw?) and perform selects on views in a different schema than the one you connected to?
Is it possible to create a connection from Postgres to Oracle (oracle_fdw?) and perform inserts on tables in the same schema as the one you connected to?
Ad 1:
Yes, certainly. Just define the foreign table as
CREATE FOREIGN TABLE view_1_r (...) SERVER ...
OPTIONS (table 'VIEW_1', schema 'USERB');
Ad 2:
Yes, certainly. Just define a foreign table on the Oracle table and insert into it. Note that bulk inserts work, but won't perform well, since there will be a round trip between PostgreSQL and Oracle for each row inserted.
Both questions indicate a general confusion between a) the Oracle user that you use to establish the connection and b) the schema of the table or view that you want to access. These things are independent: The latter is determined by the schema option of the foreign table definition, while the former is determined by the user mapping.
I am trying to create an FDW using the postgres super user credentials using the following steps -
My super user is postgres
CREATE SERVER testfdw
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '10.10.10.171', dbname 'testdb', port '5432', fetch_size '50000');
I have created a read only remote user called testdb_read_only_user on testdb and granted read only privileges to all tables and sequences and functions in there
I now create a user mapping for my local db super user postgres
CREATE USER MAPPING
FOR postgres
SERVER testfdw
OPTIONS (user 'testdb_read_only_user', password 'testpasswrd')
Then I run IMPORT FOREIGN SCHEMA public FROM SERVER testfdw INTO test_local_schema;
Now when I try to create a new foreign table into test_local_schema using the postrges super user, I am still able to create one, but the actual behaviour I expect is a permission denied error and my user mapping is based on the read only user I created on the Remote DB
Am I missing something here. Please guide.
The foreign schema test_local_schema is owned by postgres(Superuser) and PostgreSQL treating it as a local schema(on target) gives you option to create a new table, Now when you are creating a new foreign table in test_local_schema postgres will check below constraints
The name of the foreign table must be distinct from the name of any other foreign table, table, sequence, index, view, or materialized view in the same schema.
To create a foreign table, you must have USAGE(read) privilege on
the foreign server, as well as USAGE privilege on all column
types used in the table
CREATE privilege on the target schema.
As your user mapping has read-only user in source database you should be able to fetch data in order to create a foreign table in target database, however whenever you will update and insert new data it will give you error as testdb_read_only_user is not having any update/insert privileges.
https://www.postgresql.org/docs/9.5/sql-createforeigntable.html
https://www.postgresql.org/docs/9.5/sql-importforeignschema.html
I have several databases on the different PostgreSQL servers with the tables with the same columns in it
(installs_1, installs_2 and installs_3)
installs(country varchar, date datetime,paid boolean, installs int)
I want to write a function that a user could use to query across all these databases at once, how can I do it?
my query is:select country,count(*) from t1,t2
A PostgreSQL extension that offer this feature is the postgres_fdw. Here is an example of how to set it up:
First you create the extension:
CREATE EXTENSION postgres_fdw
After that you create a server pointing to the foreign postgres server
CREATE SERVER remote_postgres
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (dbname 'mydb', host 'remoteserver', port '5432');
Then an user mapping, so that an user in your current database may access the foreign database:
CREATE USER MAPPING FOR local_user
SERVER remote_postgres
OPTIONS (user 'foreign_user', password 'secret');
And finally you create a foreign table to link both tables
CREATE FOREIGN TABLE foreign_table_test
(id INT, description TEXT)
SERVER remote_postgres
OPTIONS (schema_name 'public', table_name 'table_test');
Once your table is created you can query it like you'd query a normal/local table:
SELECT * FROM foreign_table_test
Further reading:
A closer look into postgres_fdw
postgres_fdw documentation examples
Using postgres_fdw extension in Postgres 10, I am trying to shard my data.
As per below URL, I performed steps.
http://www.3manuek.com/postgresmanualsharding
Initially, I created 3 databases in my server, test1,test2 and test3. Then I did the following.
Created extension postgres_fdw.
Created 3 servers like below.
CREATE SERVER shard1_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host 'localhost',port '5432',dbname 'test1');
Mapped users for all 3 shards like below.
CREATE USER MAPPING FOR rdarukumalli SERVER shard1_main OPTIONS(user 'rdarukumalli');
Created main table and foreign tables.
CREATE TABLE main (shardKey char(2), key bigint, avalue text);
CREATE SERVER shard1_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test1');
CREATE SERVER shard2_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test2');
CREATE SERVER shard3_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test3');
Insert data into main table.
INSERT INTO main
SELECT '0' || round(random()*1+1),i.i,random()::text
FROM generate_series(1,20000) i(i) ;
After inserting data, when I verify the shards, I don't see the data is inserted into those shards. Only in main table, data lies.
But the explain command is correctly showing shards based on shard key.
Is there anything I am missing??
I have two databases that are alike, one called datastore and the other called datarestore.
datarestore is a copy of datastore which was created from a backup image. The problem is that I accidentally deleted a little too much data from datastore.
Both databases are located on different AWS instances and I typically connect to them using pgAdmin III or Python to create scripts that handle the data.
I want to get the rows that I accidentally deleted from datastore which are in datarestore into datastore. Does anyone have any idea of how this can be achieved. Both databases contain close to 1.000.000.000 rows and are on version 9.6.
I have seen some backup/import/restore options within pgAdmin III, I just don't know how they work and if they support my needs? I also thought about creating a python script, but querying my database has become pretty slow, so this seems not to be an option either.
-----------------------------------------------------
| id (serial - auto incrementing int) | - primary key
| did (varchar) |
| sensorid (int) |
| timestamp (bigint) |
| data (json) |
| db_timestamp (bigint) |
-----------------------------------------------------
If you preserved primary keys between those databases then you could create foreign tables pointing from datarestore to datastore and check what keys are missing (using for example select pk from old_table except select pk from new_table) and fetch those missing rows using the same foreign table you created. This should limit your first check for missing PK to just index only scans (+ network transfer) and then it will be index scan to fetch missing data. If you are missing only small part of it then it shouldn't take long.
If you require more detailed example then I'll update my answer.
EDIT:
Example of foreign table/server usage
Those commands need to be exuecuted on datarestore (or datastore if you choose to push data instead of pulling it).
If you don't have foreign data wrapper "installed" yet:
CREATE EXTENSION postgres_fdw;
This will create virtual server on your datarestore host. It is just some metadata pointing at foreign server:
CREATE SERVER foreign_datastore FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'foreign_hostname', dbname 'foreign_database_name',
port '5432_or_whatever_you_have_on_datastore_host');
This will tell your datarestore host what user should it connect as when using fdw on server foreign_datastore. It will be used only for your_local_role_name logged in on datarestore:
CREATE USER MAPPING FOR your_local_role_name SERVER foreign_datastore
OPTIONS (user 'foreign_username', password 'foreign_password');
You need to create schema on datarestore. It is where new foreign tables will be created.
CREATE SCHEMA schema_where_foreign_tables_will_be_created;
This will log in to remote host and create foreign tables on datarestore, pointing to tables at datastore. ONLY tables will be done this way.
No data will be copied, just structure of tables.
IMPORT FOREIGN SCHEMA foreign_datastore_schema_name_goes_here
FROM SERVER foreign_datastore INTO schema_where_foreign_tables_will_be_created;
This will return list of id that are missing in your datarestore database for this table
SELECT id FROM foreign_datastore_schema_name_goes_here.table_a
EXCEPT
SELECT id FROM datarestore_schema.table_a
You can either store them in temp table (CREATE TABLE table_a_missing_pk AS [query from above here]
Or use them right away:
INSERT INTO datarestore_schema.table_a (id, did, sensorid, timestamp, data, db_timestamp)
SELECT id, did, sensorid, timestamp, data, db_timestamp
FROM foreign_datastore_schema_name_goes_here.table_a
WHERE id = ANY((
SELECT array_agg(id)
FROM (
SELECT id FROM foreign_datastore_schema_name_goes_here.table_a
EXCEPT
SELECT id FROM datarestore_schema.table_a
) sub
)::int[])
From my tests, this should push-down (meaning send to remote host) something like that:
Remote SQL: SELECT id, did, sensorid, timestamp, data, db_timestamp
FROM foreign_datastore_schema_name_goes_here.table_a WHERE ((id = ANY ($1::integer[])))
You can make sure it does by running explain verbose on your full query to see what plan it will execute. You should see Remote SQL in there.
In case it does not work as expected, you can instead create temp table as mentioned earlier and make sure that this temp table is on datastore host.
Alternative approach would be to create foreign server on datastore pointing to datarestore and push data from your old database to new one (you can insert into foreign tables). This way you won't have to worry about list of id not being pushed down to datastore and instead fetching all data and filtering them afterwards (with would be extremely slow).