Why would postgres refuse to use a foreign key? Here is my query, the join is on a foreign key.
EXPLAIN
SELECT *
FROM listings_searchresult
JOIN listings_searchquery ON listings_searchresult.search_query_id = listings_searchquery.id
Hash Cond: (listings_searchresult.search_query_id = listings_searchquery.id)
-> Seq Scan on listings_searchresult (cost=0.00..4898345.08 rows=83607008 width=1129)
-> Hash (cost=570499.88..570499.88 rows=20226788 width=109)
-> Seq Scan on listings_searchquery (cost=0.00..570499.88 rows=20226788 width=109)
Why would postgres not use the foreign key? I delete it and readded it in case it was corrupted, still not working. Can I somehow force postgres to use this?
Unless one of the tables is very small, reading the whole tables is the most efficient technique to process such a query.
With a nested loop join, which is what you envision, PostgreSQL would have to scan an index on listings_searchresult 20 million times.
Using a hash join as it does, PostgreSQL builds a hash table in memory from the smaller table and probes that hash table for each row in the bigger table, which will perform better.
Joining two big tables without an additional WHERE condition is always going to be slow and can potentially produce a large result set.
Related
So I have a (logged) table with two columns A, B, containing text.
They basically contain the same type of information, it's just two columns because of where the data came from.
I wanted to have a table of all unique values (so I made the column be the primary key), not caring about the column. But when I asked postgres to do
insert into new_table(value) select A from old_table on conflict (value) do nothing; (and later on same thing for column B)
it used 1 cpu core, and only read from my SSD with about 5 MB/s. I stopped it after a couple of hours.
I suspected that it might be because of the b-tree being slow and so I added a hashindex on the only attribute in my new table. But it's still using 1 core to the max and reading from the ssd at only 5 MB/s per second. My java program can hashset that at at least 150 MB/s, so postgres should be way faster than 5 MB/s, right? I've analyzed my old table and I made my new table unlogged for faster inserts and yet it still uses 1 core and reads extremely slowly.
How to fix this?
EDIT: This is the explain to the above query. Seems like postgres is using the b-tree it created for the primary key instead of my (much faster, isn't it??) Hash index.
Insert on users (cost=0.00..28648717.24 rows=1340108416 width=14)
Conflict Resolution: NOTHING
Conflict Arbiter Indexes: users_pkey
-> Seq Scan on games (cost=0.00..28648717.24 rows=1340108416 width=14)
The ON CONFLICT mechanism is primarily for resolving concurrency-induced conflicts. You can use it in a "static" case like this, but other methods will be more efficient.
Just insert only distinct values in the first place:
insert into new_table(value)
select A from old_table union
select B from old_table
For increased performance, don't add the primary key until after the table is populated. And set work_mem to the largest value you credibly can.
My java program can hashset that at at least 150 MB/s,
That is working with the hashset entirely in memory. PostgreSQL indexes are disk-based structures. They do benefit from caching, but that only goes so far and depends on hardware and settings you haven't told us about.
Seems like postgres is using the b-tree it created for the primary key instead of my (much faster, isn't it??) Hash index.
It can only use the index which defines the constraint, which is the btree index, as hash indexes cannot support primary key constraints. You could define an EXCLUDE constraint using the hash index, but that would just make it slower yet. And in general, hash indexes are not "much faster" than btree indexes in PostgreSQL.
Let's say we have a table table1 with field1 INT ENCODE ZSTD, and we added interleaved sort key on field1.
But when I do the query select * from table1 where field1=123;, I still see a sequential scan on the whole table which I suppose should be a sub scan on the table.
Do I have some misunderstanding about sort key?
1) From what you tell, you don't need an interleaved sort key because you have just one column you're interested in. You need an interleaved sort key when you want multiple columns to be equally important because you want to run both where col1=123 and where col2=123 kind of queries. This provides benefit for large tables.
2) Compressing your sort key column is considered a bad practice. Proof from Amazon: the first column in a compound sort key should not be encoded (one column key is the same). The entire article is actually useful, read it and you won't regret
3) When sort key is configured and data is populated it's better to run vacuum and analyze commands to make sure rows are sorted accordingly to sort key and table statistics are updated.
I'm looking at the performance of some queries that I'm doing in Redshift and noticed something that I can't quite find in the documentation.
I created two tables that have a join key between them (about 10K rows in the child table).
For the parent table, let's call it A, I have a primary key that I've declared to be the distkey and sort key for the table. Let's call this id.
For the child table B, I've made a foreign key field, parent_id that references A.id. parent_id has been declared as the distkey for table B. Table B also has a primary key, id that I've defined. I've created an interleaved sort key on table B for (parent_id,id).
When I try to do an explain joining the two tables, I will always get a Hash Join. If I recreate table B with a normal compound sort key, I will always get a Merge Join.
When I look at the stats of the tables, I don't see any skews that are out of line.
My question is, will Redshift always use Hash Joins with interleaved sort keys or is there something I'm doing wrong?
EDIT - The order of the interleaved sort keys in Table B is actually (parent_id, id). I wrote it above incorrectly. I've updated the above to be clear now.
From my understanding:
A merge join can be used when both tables are sorted on the join column, which is very efficient -- a bit like closing a zipper, where both sides "fit into" each other.
A hash join is less efficient because it needs to do lookups via hashes to find matching values.
As you pointed out, if the tables are sorted using a normal compound key, then both tables are sorted by the join column.
In an interleaved join, however, values are not guaranteed to be sorted within each column.
The documentation for Interleaved Keys says:
An interleaved sort gives equal weight to each column, or subset of columns, in the sort key. If multiple queries use different columns for filters, then you can often improve performance for those queries by using an interleaved sort style. When a query uses restrictive predicates on secondary sort columns, interleaved sorting significantly improves query performance as compared to compound sorting.
However, it does not mean that all columns are sorted (as they are with a Compound sort). Rather, it gives a generally good mix of sorting, so that sorts on any column work generally well. Therefore, each column is not necessarily fully sorted, hence the need for a hash join.
The blog post Quickly Filter Data in Amazon Redshift Using Interleaved Sorting tries to explain how the data is stored when using interleaved sorting.
I'm running two different postgres 9.3 instances (one for production, one for development/testing). I want to copy a subset of a table in production into development.
Let's say then that the table I want to copy is defined as
CREATE TABLE users (user_id varchar PRIMARY KEY, other_stuff varchar);
and the subset I want to copy is all the users who have user_id in a cached table (on production), which is a much smaller table than the users table
CREATE TABLE user_id_subset (user_id varchar PRIMARY KEY);
I've set up some foreign tables on my development db to access these two tables (named foreign_users, and foreign_user_id_subset respectively), and I want to do a query like:
INSERT INTO development_users (user_id, other_stuff)
SELECT user_id, other_stuff FROM foreign_users f
WHERE EXISTS (
SELECT 1 FROM foreign_user_id_subset ss
WHERE ss.user_id=f.user_id)
This query works, but I'm worried about performance. The result of an explain gives me something like this:
'Insert on development_users (cost=262.01..284.09 rows=138 width=272)'
' -> Hash Join (cost=262.01..284.09 rows=138 width=272)'
' Hash Cond: ((f.user_id)::text = (cache.user_id)::text)'
' -> Foreign Scan on foreign_users f (cost=100.00..118.28 rows=276 width=272)'
' -> Hash (cost=159.52..159.52 rows=200 width=32)'
' -> HashAggregate (cost=157.52..159.52 rows=200 width=32)'
' -> Foreign Scan on foreign_user_id_subset (cost=100.00..153.86 rows=1462 width=32)'
What I think is happening is that my development db sends the request to my production db, which creates the temp hash of foreign_user_id_subset (user_id_subset on production) and does the hash check on production. This way, the only thing that gets sent across the wire (between the databases) is the initial request, and then the result of the select query. Is this true?
An alternative idea would be to create a 'temp' (can't be a true TEMP table b/c I need a foreign table) of the result of this request on my production database, and then build a foreign table and just do a SELECT * from development on the foreign table.
(It should be noted that my production database is a much pricier/performant RDS instance than my development database)
To answer my own question:
I implemented the alternative idea described above: generating the table in production containing only the results of the EXISTS query, and then created a foreign table in development to reference that table. Then to create the subset table, I just did an 'INSERT INTO ...SELECT * FROM ...'.
The timing was dramatically faster using this than the original method posted.
Time to create table on production:
Total runtime: 204.838 ms
Time to insert into development database: Total runtime: 1564.444 ms
And to do the original method:
it's unacceptably slow, taking > 10 minutes to achieve the same result as above (I didn't even wait for the explain analyze to finish)
I don't know a better way to determine lower-level instructions past what the planner provides, but I'm lead to believe that the original method performs a sequential scan (made by repeatedly scanning chunks of the foreign table) and performs the hash comparison on the development database.
I'm not trying to restart the UUID vs serial integer key debate. I know there are valid points to either side. I'm using UUID's as the primary key in several of my tables.
Column type: "uuidKey" text NOT NULL
Index: CREATE UNIQUE INDEX grand_pkey ON grand USING btree ("uuidKey")
Primary Key Constraint: ADD CONSTRAINT grand_pkey PRIMARY KEY ("uuidKey");
Here is my first question; with PostgreSQL 9.4 is there any performance benefit to setting the column type to UUID?
The documentation http://www.postgresql.org/docs/9.4/static/datatype-uuid.html describes UUID's, but is there any benefit aside from type safety for using this type instead of text type? In the character types documentation it indicates that char(n) would not have any advantage over text in PostgreSQL.
Tip: There is no performance difference among these three types, apart
from increased storage space when using the blank-padded type, and a
few extra CPU cycles to check the length when storing into a
length-constrained column. While character(n) has performance
advantages in some other database systems, there is no such advantage
in PostgreSQL; in fact character(n) is usually the slowest of the
three because of its additional storage costs. In most situations text
or character varying should be used instead.
I'm not worried about disk space, I'm just wondering if it's worth my time benchmarking UUID vs text column types?
Second question, hash vs b-tree indexes. No sense in sorting UUID keys so would b-tree have any other advantages over hash index?
We had a table with about 30k rows that (for a specific unrelated architectural reason) had UUIDs stored in a text field and indexed. I noticed that the query perf was slower than I'd have expected. I created a new UUID column, copied in the text uuid primary key and compared below. 2.652ms vs 0.029ms. Quite a difference!
-- With text index
QUERY PLAN
Index Scan using tmptable_pkey on tmptable (cost=0.41..1024.34 rows=1 width=1797) (actual time=0.183..2.632 rows=1 loops=1)
Index Cond: (primarykey = '755ad490-9a34-4c9f-8027-45fa37632b04'::text)
Planning time: 0.121 ms
Execution time: 2.652 ms
-- With a uuid index
QUERY PLAN
Index Scan using idx_tmptable on tmptable (cost=0.29..2.51 rows=1 width=1797) (actual time=0.012..0.013 rows=1 loops=1)
Index Cond: (uuidkey = '755ad490-9a34-4c9f-8027-45fa37632b04'::uuid)
Planning time: 0.109 ms
Execution time: 0.029 ms
A UUID is a 16 bytes value. The same as text is a 32 bytes value. The storage sizes are:
select
pg_column_size('a0eebc999c0b4ef8bb6d6bb9bd380a11'::text) as text_size,
pg_column_size('a0eebc999c0b4ef8bb6d6bb9bd380a11'::uuid) as uuid_size;
text_size | uuid_size
-----------+-----------
36 | 16
Smaller tables lead to faster operations.