Using postgres_fdw extension in Postgres 10, I am trying to shard my data.
As per below URL, I performed steps.
http://www.3manuek.com/postgresmanualsharding
Initially, I created 3 databases in my server, test1,test2 and test3. Then I did the following.
Created extension postgres_fdw.
Created 3 servers like below.
CREATE SERVER shard1_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host 'localhost',port '5432',dbname 'test1');
Mapped users for all 3 shards like below.
CREATE USER MAPPING FOR rdarukumalli SERVER shard1_main OPTIONS(user 'rdarukumalli');
Created main table and foreign tables.
CREATE TABLE main (shardKey char(2), key bigint, avalue text);
CREATE SERVER shard1_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test1');
CREATE SERVER shard2_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test2');
CREATE SERVER shard3_main FOREIGN DATA WRAPPER postgres_fdw
OPTIONS(host ‘localhost',port '5432',dbname 'test3');
Insert data into main table.
INSERT INTO main
SELECT '0' || round(random()*1+1),i.i,random()::text
FROM generate_series(1,20000) i(i) ;
After inserting data, when I verify the shards, I don't see the data is inserted into those shards. Only in main table, data lies.
But the explain command is correctly showing shards based on shard key.
Is there anything I am missing??
Related
Let's assume that I have db1 and db2, two databases. I would like to perform a command of the like of
update db2.person p2
set p2.name = p1.name
from db1.person p1
where p1.id = p2.id;
This is possible in MySQL without any problems. I have great difficulty achieving it in PostgreSQL.
What I have tried:
create extension postgres_fdw;
create server theservername
foreign data wrapper postgres_fdw
options(host 'localhost', dbname 'thedbname', port '5432');
create user mapping for theuser
server theservername
options(user 'theusername', password 'thepassword');
And here I'm stuck, I don't know how to proceed. None of these troubles exist in MySQL. How can I overcome them in PostgreSQL?
Steps are following:
Step - 1: Create Extension
create extension postgres_fdw;
Step - 2: Create Server
create server theservername
foreign data wrapper postgres_fdw
options(host 'localhost', dbname 'thedbname', port '5432');
Step - 3: Create Foreign User Mapping for the server
create user mapping for theuser
server theservername
options(user 'theusername', password 'thepassword');
Step - 4: Create Foreign Table with same structure as in another DB
create foreign table "schema_name"."local_table_name"
(
id_ int;
...
-- field list same as foreign table in other db
)
server theservername
options(SCHEMA_NAME 'foreign_schema', TABLE_NAME 'foreign_name');
Now you can use local_table_name in your query just as local table. It will do all operations on remote db.
Your update query can be written like below:
update local_table_name p2
set name = p1.name
from person p1
where p1.id = p2.id;
I have several databases on the different PostgreSQL servers with the tables with the same columns in it
(installs_1, installs_2 and installs_3)
installs(country varchar, date datetime,paid boolean, installs int)
I want to write a function that a user could use to query across all these databases at once, how can I do it?
my query is:select country,count(*) from t1,t2
A PostgreSQL extension that offer this feature is the postgres_fdw. Here is an example of how to set it up:
First you create the extension:
CREATE EXTENSION postgres_fdw
After that you create a server pointing to the foreign postgres server
CREATE SERVER remote_postgres
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (dbname 'mydb', host 'remoteserver', port '5432');
Then an user mapping, so that an user in your current database may access the foreign database:
CREATE USER MAPPING FOR local_user
SERVER remote_postgres
OPTIONS (user 'foreign_user', password 'secret');
And finally you create a foreign table to link both tables
CREATE FOREIGN TABLE foreign_table_test
(id INT, description TEXT)
SERVER remote_postgres
OPTIONS (schema_name 'public', table_name 'table_test');
Once your table is created you can query it like you'd query a normal/local table:
SELECT * FROM foreign_table_test
Further reading:
A closer look into postgres_fdw
postgres_fdw documentation examples
I have two databases that are alike, one called datastore and the other called datarestore.
datarestore is a copy of datastore which was created from a backup image. The problem is that I accidentally deleted a little too much data from datastore.
Both databases are located on different AWS instances and I typically connect to them using pgAdmin III or Python to create scripts that handle the data.
I want to get the rows that I accidentally deleted from datastore which are in datarestore into datastore. Does anyone have any idea of how this can be achieved. Both databases contain close to 1.000.000.000 rows and are on version 9.6.
I have seen some backup/import/restore options within pgAdmin III, I just don't know how they work and if they support my needs? I also thought about creating a python script, but querying my database has become pretty slow, so this seems not to be an option either.
-----------------------------------------------------
| id (serial - auto incrementing int) | - primary key
| did (varchar) |
| sensorid (int) |
| timestamp (bigint) |
| data (json) |
| db_timestamp (bigint) |
-----------------------------------------------------
If you preserved primary keys between those databases then you could create foreign tables pointing from datarestore to datastore and check what keys are missing (using for example select pk from old_table except select pk from new_table) and fetch those missing rows using the same foreign table you created. This should limit your first check for missing PK to just index only scans (+ network transfer) and then it will be index scan to fetch missing data. If you are missing only small part of it then it shouldn't take long.
If you require more detailed example then I'll update my answer.
EDIT:
Example of foreign table/server usage
Those commands need to be exuecuted on datarestore (or datastore if you choose to push data instead of pulling it).
If you don't have foreign data wrapper "installed" yet:
CREATE EXTENSION postgres_fdw;
This will create virtual server on your datarestore host. It is just some metadata pointing at foreign server:
CREATE SERVER foreign_datastore FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'foreign_hostname', dbname 'foreign_database_name',
port '5432_or_whatever_you_have_on_datastore_host');
This will tell your datarestore host what user should it connect as when using fdw on server foreign_datastore. It will be used only for your_local_role_name logged in on datarestore:
CREATE USER MAPPING FOR your_local_role_name SERVER foreign_datastore
OPTIONS (user 'foreign_username', password 'foreign_password');
You need to create schema on datarestore. It is where new foreign tables will be created.
CREATE SCHEMA schema_where_foreign_tables_will_be_created;
This will log in to remote host and create foreign tables on datarestore, pointing to tables at datastore. ONLY tables will be done this way.
No data will be copied, just structure of tables.
IMPORT FOREIGN SCHEMA foreign_datastore_schema_name_goes_here
FROM SERVER foreign_datastore INTO schema_where_foreign_tables_will_be_created;
This will return list of id that are missing in your datarestore database for this table
SELECT id FROM foreign_datastore_schema_name_goes_here.table_a
EXCEPT
SELECT id FROM datarestore_schema.table_a
You can either store them in temp table (CREATE TABLE table_a_missing_pk AS [query from above here]
Or use them right away:
INSERT INTO datarestore_schema.table_a (id, did, sensorid, timestamp, data, db_timestamp)
SELECT id, did, sensorid, timestamp, data, db_timestamp
FROM foreign_datastore_schema_name_goes_here.table_a
WHERE id = ANY((
SELECT array_agg(id)
FROM (
SELECT id FROM foreign_datastore_schema_name_goes_here.table_a
EXCEPT
SELECT id FROM datarestore_schema.table_a
) sub
)::int[])
From my tests, this should push-down (meaning send to remote host) something like that:
Remote SQL: SELECT id, did, sensorid, timestamp, data, db_timestamp
FROM foreign_datastore_schema_name_goes_here.table_a WHERE ((id = ANY ($1::integer[])))
You can make sure it does by running explain verbose on your full query to see what plan it will execute. You should see Remote SQL in there.
In case it does not work as expected, you can instead create temp table as mentioned earlier and make sure that this temp table is on datastore host.
Alternative approach would be to create foreign server on datastore pointing to datarestore and push data from your old database to new one (you can insert into foreign tables). This way you won't have to worry about list of id not being pushed down to datastore and instead fetching all data and filtering them afterwards (with would be extremely slow).
I have master-slave architecture. On the slave I have an Oracle database with two schemas, e.g. TEST1 and TEST2.
I have all objects (e.g. EMPLOYEES) stored in schema TEST1, and user TEST1 (or admin) has given read only privileges on TEST1.EMPLOYEES to TEST2, so when I use TEST1.EMPLOYEES in a query on the Oracle database I can access its data.
How can I implement the same using Oracle foreign data wrapper in postgres 9.5 because I have credentials for TEST2 and not TEST1?
When I try to access the foreign table it give an error saying that TEST2.EMPLOYEES does not exist.
You can easily do that if you define the user mapping with the credentials of user TEST2 and the foreign table with the schema option, i.e.
CREATE FOREIGN TABLE ... OPTIONS (schema 'TEST1', table 'EMPLOYEES');
Is it possible to create a foreign table, using Postgres Foreign Data Wrapper, that points to a view instead of a table?
Yes, it is possible!
The following query worked perfectly:
CREATE FOREIGN TABLE facts(name character varying(255))
SERVER my_server
OPTIONS (table_name 'facts');
Where facts is a view in my_server instead of a table.
Recently I had to do the same thing and here are the steps that worked for me. All these commands are run on the local postgreSQL DB.
CREATE EXTENSION postgres_fdw;
CREATE SERVER remote_server_name
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '10.10.10.10', port '5432', dbname 'remote_db_name');
CREATE USER MAPPING FOR local_user_name
SERVER remote_server_name
OPTIONS (user 'remote_user', password 'remote_password');
CREATE FOREIGN TABLE local_table_name (
id NUMERIC NOT NULL,
row TEXT,
another_row INTEGER,
whatever_row TEXT
)
SERVER remote_server_name
OPTIONS (schema_name 'public', table_name 'remote_table_name');
I have the same question.
In pgadmin4 for postgresql-11, if use GUI Command: Create -> Foreign Table...
on table, it works; but on view, it does't works, you will get a empty table.
for view, i use this code, it works:
IMPORT FOREIGN SCHEMA remote_schema_name
LIMIT TO (remote_view_name)
FROM SERVER remote_host_map_name INTO local_shema_name;
The reason is, for table, pgadmin4 can create columns same as remote table in constract SQL statement, but for view, it create no columns in constract SQL statement.