I'm running two different postgres 9.3 instances (one for production, one for development/testing). I want to copy a subset of a table in production into development.
Let's say then that the table I want to copy is defined as
CREATE TABLE users (user_id varchar PRIMARY KEY, other_stuff varchar);
and the subset I want to copy is all the users who have user_id in a cached table (on production), which is a much smaller table than the users table
CREATE TABLE user_id_subset (user_id varchar PRIMARY KEY);
I've set up some foreign tables on my development db to access these two tables (named foreign_users, and foreign_user_id_subset respectively), and I want to do a query like:
INSERT INTO development_users (user_id, other_stuff)
SELECT user_id, other_stuff FROM foreign_users f
WHERE EXISTS (
SELECT 1 FROM foreign_user_id_subset ss
WHERE ss.user_id=f.user_id)
This query works, but I'm worried about performance. The result of an explain gives me something like this:
'Insert on development_users (cost=262.01..284.09 rows=138 width=272)'
' -> Hash Join (cost=262.01..284.09 rows=138 width=272)'
' Hash Cond: ((f.user_id)::text = (cache.user_id)::text)'
' -> Foreign Scan on foreign_users f (cost=100.00..118.28 rows=276 width=272)'
' -> Hash (cost=159.52..159.52 rows=200 width=32)'
' -> HashAggregate (cost=157.52..159.52 rows=200 width=32)'
' -> Foreign Scan on foreign_user_id_subset (cost=100.00..153.86 rows=1462 width=32)'
What I think is happening is that my development db sends the request to my production db, which creates the temp hash of foreign_user_id_subset (user_id_subset on production) and does the hash check on production. This way, the only thing that gets sent across the wire (between the databases) is the initial request, and then the result of the select query. Is this true?
An alternative idea would be to create a 'temp' (can't be a true TEMP table b/c I need a foreign table) of the result of this request on my production database, and then build a foreign table and just do a SELECT * from development on the foreign table.
(It should be noted that my production database is a much pricier/performant RDS instance than my development database)
To answer my own question:
I implemented the alternative idea described above: generating the table in production containing only the results of the EXISTS query, and then created a foreign table in development to reference that table. Then to create the subset table, I just did an 'INSERT INTO ...SELECT * FROM ...'.
The timing was dramatically faster using this than the original method posted.
Time to create table on production:
Total runtime: 204.838 ms
Time to insert into development database: Total runtime: 1564.444 ms
And to do the original method:
it's unacceptably slow, taking > 10 minutes to achieve the same result as above (I didn't even wait for the explain analyze to finish)
I don't know a better way to determine lower-level instructions past what the planner provides, but I'm lead to believe that the original method performs a sequential scan (made by repeatedly scanning chunks of the foreign table) and performs the hash comparison on the development database.
Related
I have a Postgres table named: services, and it has columns called id, mac_addr, dns_name, hash, and it is partitioned based on mac_addr, so the partition tables names look like: services_3eeeea123e3 and so on. And there are around 20K partition based on mac_addrs
Q1: there was no index created when the tables were created. so now, when I am trying to add an index CREATE INDEX idx_services_id on services (id), it throws an error ERROR: cannot create an index on partitioned table "services"
But I am able to add indexes to individual partitioned tables CREATE INDEX idx_services_3eeeea123e3 on services_3eeeea123e3 (id).
So do I have to create an index on each partition table now? Is there a way to create an index on the base table(services) itself, which will automatically create an index on each partition table?
Q2: When I run a select query, it is fast when I use the direct partition table; however, using the base table is very slow. Any idea what could be the reason.
Fast: SELECT id, dns_name, hash from services_3eeeea123e3 where id='123232'
very slow: SELECT id,dns_name, hash from services where mac_addr='3eeeea123e3' and id='123232'
Why would postgres refuse to use a foreign key? Here is my query, the join is on a foreign key.
EXPLAIN
SELECT *
FROM listings_searchresult
JOIN listings_searchquery ON listings_searchresult.search_query_id = listings_searchquery.id
Hash Cond: (listings_searchresult.search_query_id = listings_searchquery.id)
-> Seq Scan on listings_searchresult (cost=0.00..4898345.08 rows=83607008 width=1129)
-> Hash (cost=570499.88..570499.88 rows=20226788 width=109)
-> Seq Scan on listings_searchquery (cost=0.00..570499.88 rows=20226788 width=109)
Why would postgres not use the foreign key? I delete it and readded it in case it was corrupted, still not working. Can I somehow force postgres to use this?
Unless one of the tables is very small, reading the whole tables is the most efficient technique to process such a query.
With a nested loop join, which is what you envision, PostgreSQL would have to scan an index on listings_searchresult 20 million times.
Using a hash join as it does, PostgreSQL builds a hash table in memory from the smaller table and probes that hash table for each row in the bigger table, which will perform better.
Joining two big tables without an additional WHERE condition is always going to be slow and can potentially produce a large result set.
So I have a (logged) table with two columns A, B, containing text.
They basically contain the same type of information, it's just two columns because of where the data came from.
I wanted to have a table of all unique values (so I made the column be the primary key), not caring about the column. But when I asked postgres to do
insert into new_table(value) select A from old_table on conflict (value) do nothing; (and later on same thing for column B)
it used 1 cpu core, and only read from my SSD with about 5 MB/s. I stopped it after a couple of hours.
I suspected that it might be because of the b-tree being slow and so I added a hashindex on the only attribute in my new table. But it's still using 1 core to the max and reading from the ssd at only 5 MB/s per second. My java program can hashset that at at least 150 MB/s, so postgres should be way faster than 5 MB/s, right? I've analyzed my old table and I made my new table unlogged for faster inserts and yet it still uses 1 core and reads extremely slowly.
How to fix this?
EDIT: This is the explain to the above query. Seems like postgres is using the b-tree it created for the primary key instead of my (much faster, isn't it??) Hash index.
Insert on users (cost=0.00..28648717.24 rows=1340108416 width=14)
Conflict Resolution: NOTHING
Conflict Arbiter Indexes: users_pkey
-> Seq Scan on games (cost=0.00..28648717.24 rows=1340108416 width=14)
The ON CONFLICT mechanism is primarily for resolving concurrency-induced conflicts. You can use it in a "static" case like this, but other methods will be more efficient.
Just insert only distinct values in the first place:
insert into new_table(value)
select A from old_table union
select B from old_table
For increased performance, don't add the primary key until after the table is populated. And set work_mem to the largest value you credibly can.
My java program can hashset that at at least 150 MB/s,
That is working with the hashset entirely in memory. PostgreSQL indexes are disk-based structures. They do benefit from caching, but that only goes so far and depends on hardware and settings you haven't told us about.
Seems like postgres is using the b-tree it created for the primary key instead of my (much faster, isn't it??) Hash index.
It can only use the index which defines the constraint, which is the btree index, as hash indexes cannot support primary key constraints. You could define an EXCLUDE constraint using the hash index, but that would just make it slower yet. And in general, hash indexes are not "much faster" than btree indexes in PostgreSQL.
I'm running into something I cannot explain and I have been googling for a few days now and have not yet found the cause for my "problem" with the PostgresQL scheduler causing a (relatively simple) query to take massive amounts of time.
Let's start from the top (I've tried to remove as much useless information as possible so the tables may look pointless but trust me, they're not):
I have the following schema:
CREATE TABLE ct_log (
ID integer,
CONSTRAINT ctl_pk
PRIMARY KEY (ID)
);
CREATE TABLE ct_log_entry (
CERTIFICATE_ID bigint NOT NULL,
ENTRY_ID bigint NOT NULL,
ENTRY_TIMESTAMP timestamp NOT NULL,
CT_LOG_ID integer NOT NULL,
CONSTRAINT ctle_ctl_fk
FOREIGN KEY (CT_LOG_ID)
REFERENCES ct_log(ID)
) PARTITION BY RANGE (ENTRY_TIMESTAMP);
-- I will not repeat this one 7 times, but there are partition for each year from 2013-2020:
CREATE TABLE ct_log_entry_2020 PARTITION OF ct_log_entry
FOR VALUES FROM ('2020-01-01T00:00:00'::timestamp) TO ('2021-01-01T00:00:00'::timestamp);
CREATE INDEX ctle_c ON ct_log_entry (CERTIFICATE_ID);
CREATE INDEX ctle_e ON ct_log_entry (ENTRY_ID);
CREATE INDEX ctle_t ON ct_log_entry (ENTRY_TIMESTAMP);
CREATE INDEX ctle_le ON ct_log_entry (CT_LOG_ID, ENTRY_ID DESC);
(in case you are curious about the full schema: https://github.com/crtsh/certwatch_db/blob/master/sql/create_schema.sql)
And this is the query I am trying to run:
SELECT ctl.ID, latest.entry_id
FROM ct_log ctl
LEFT JOIN LATERAL (
SELECT coalesce(max(entry_id), -1) entry_id
FROM ct_log_entry ctle
WHERE ctle.ct_log_id = ctl.id
) latest ON TRUE;
For the people that know https://crt.sh this might look familiar because this is indeed the schema from crt.sh. This makes it a bit interesting since crt.sh provides public PostgresQL access allowing me to compare query plans between my own server and theirs.
My server query plan (~700s): https://explain.depesz.com/s/ZKkt
Public crt.sh query plan (~3ms): https://explain.depesz.com/s/01Ht
This difference is quit noticeable (:sad_smile:) but I'm not sure why because as far as I know I have the correct indexes for this to be very fast and the same indexes as the crt.sh server.
It looks like my instance is using a backwards index scan instead of a index only scan for the largest 2 partitions. This was not always the case and previously it execute using the same query plan as the crt.sh instance but for some reason it decided to stop doing that.
(This is the amount of data in those tables in case it's not clear from the query plans: https://d.bouma.dev/wUjdXJXk1OzF. I cannot see how much is in the crt.sh database because they don't provide access to the individual partitions)
Now onto the list of thing I've tried:
ANALYZE the ct_log_entry (and ct_log_entry_* tables created by the partitioning)
VACUUM ANALYZE the ct_log_entry (and ct_log_entry_* tables created by the partitioning)
VACUUM FULL the ct_log_entry (and ct_log_entry_* tables created by the partitioning)
Dropping the ctle_le index and recreating it again (this worked once for me giving me a few hours of great performance until I imported more data and it went with the backwards scan again)
REINDEX INDEX the ctle_le index on each ct_log_entry_* table
SET random_page_cost = x;, tried 1, 1.1, 4 and 5 (according to many SO answers and blog posts)
The only thing I notice that are different is that crt.sh is running PostgresQL 12.1 and I'm running 12.3, but as far as I can tell that shouldn't have any impact.
Also before you say, "yes well, but you cannot run this amount of data on your laptop", the server I'm running is a dedicated box with 32 available threads and 128GB RAM and running a RAID 5 with 8 2TB Samsung EVO 860 drives on hardware RAID (yes I know this is bad if a drive fails, that's another issue I'll deal with later but the read performance should be excellent). I don't know what crt.sh is running for hardware but since I only have a fraction of the data imported I don't see my hardware being the issue here (yet).
I've also "tuned" my config using the guide here: https://pgtune.leopard.in.ua/#/.
Happy to provide more info where needed but hoping someone can point me to a flaw and/or provide a solution to resolve the problem and show PostgresQL how to use the optimal path!
suppose I have a query like
select * from remote_table
join local_table using(common_key)
where remote_table is a FOREIGN TABLE with postgres_fdw and local_table is a regular table.
local_table is small (100 rows) and remote_table is large (millions of rows).
It looks like the remote table is pulled in its entirety and joined locally, when it would be more efficient to ship the smaller table to the remote server and join remotely.
Is there a way to get postgres_fdw to do that?
You cannot do that with a join, since joins between tables on different servers are always executed locally.
What you could try is something like:
SELECT *
FROM (SELECT *
FROM remote_table
WHERE common_key IN (SELECT common_key FROM local_table)
) a
JOIN local_table USING (common_key);
I did not test it, so I am not sure if it will work, but the idea is to use a condition for the foreign table scan that can be pushed down and reduces the amount of data fetched as much as possible.
Have you tried deploying the local data into a temp table on the foreign server then joining it into the foreign table? Not sure of your process or if this would be efficient for you or not.