I am using a compound index on a table with more than 13 million records.
The index order is (center_code, created_on, status). The center_code and status both are varchar(100) not NULL and created_on is timestamp without time zone.
I read somewhere that order of indexes matter in a compound index. We have to check for number of unique values and put the one with the highest number of unique values at the first place in compound index.
The center_code can have 4000 distinct values.
The status can have 5 distinct values.
The min value of created_on is 2017-12-12 02:00:49.465317+00.
The question is what can be the number of unique values for created_on?
Should I put it first in the compound index?
Indexing on date column works on date basis, hour basis or second basis.
The problem is:
A simple SELECT query is taking more than 500 ms which is using just this compound index and nothing else.
Indexes on table:
Indexes:
"pa_key" PRIMARY KEY, btree (id)
"pa_uniq" UNIQUE CONSTRAINT, btree (wbill)
"pa_center_code_created_on_status_idx_new" btree (center_code, created_on, status)
The query is:
EXPLAIN ANALYSE
SELECT "pa"."wbill"
FROM "pa"
WHERE ("pa"."center_code" = 'IND110030AAC'
AND "pa"."status" IN ('Scheduled')
AND "pa"."created_on" >= '2018-10-10T00:00:00+05:30'::timestamptz);
Query Plan:
Index Scan using pa_center_code_created_on_status_idx_new on pa (cost=0.69..3769.18 rows=38 width=13) (actual time=5.592..15.526 rows=78 loops=1)
Index Cond: (((center_code)::text = 'IND110030AAC'::text) AND (created_on >= '2018-10-09 18:30:00+00'::timestamp with time zone) AND ((status)::text = 'Scheduled'::text))
Planning time: 1.156 ms
Execution time: 519.367 ms
Any help would be highly appreciated.
The index scan condition reads
(((center_code)::text = 'IND110030AAC'::text) AND
(created_on >= '2018-10-09 18:30:00+00'::timestamp with time zone) AND
((status)::text = 'Scheduled'::text))
but the index scan itself is only over (center_code, created_on), while the condition on status is applied as a filter.
Unfortunately this is not visible from the execution plan, but it follows from the following rule:
An index scan will only use conditions if the rows satisfying the conditions are next to each other in the index.
Let's consider this example (in index order):
center_code | created_on | status
--------------+---------------------+-----------
IND110030AAC | 2018-10-09 00:00:00 | Scheduled
IND110030AAC | 2018-10-09 00:00:00 | Xtra
IND110030AAC | 2018-10-10 00:00:00 | New
IND110030AAC | 2018-10-10 00:00:00 | Scheduled
IND110030AAC | 2018-10-11 00:00:00 | New
IND110030AAC | 2018-10-11 00:00:00 | Scheduled
You will see that the query needs the 4th and 6th row.
PostgreSQL cannot scan the index with all three conditions, because the required rows are not next to each other. It will have to scan only with the first two conditions, because all rows satisfying those are right next to each other.
Your rule for multi-column indexes is wrong. The columns at the left of the index have to be the ones where = is used as comparison operator in the conditions.
The perfect index would be one on (center_code, status, created_on).
One of the tips that I have learned from working is that when you created compound idx, the column with condition (=) should be priority and other conditions like (>, <, >=, <=, IN) will follow after.
Related
I have two tables exchange_rate (100 Thousand Rows) and paid_date_t (9 million rows) with below structure.
"exchange_rate"
Column | Type | Collation | Nullable | Default
-----------------------------+--------------------------+-----------+----------+---------
valid_from | timestamp with time zone | | |
valid_until | timestamp with time zone | | |
currency | text | | |
Indexes:
"exchange_rate_unique_valid_from_currency_key" UNIQUE, btree (valid_from, currency)
"exchange_rate_valid_from_gist_idx" gist (valid_from)
"exchange_rate_valid_from_until_currency_gist_idx" gist (valid_from, valid_until, currency)
"exchange_rate_valid_from_until_gist_idx" gist (valid_from, valid_until)
"exchange_rate_valid_until_gist_idx" gist (valid_until)
"paid_date_t"
Column | Type | Collation | Nullable | Default
-------------------+-----------------------------+-----------+----------+---------
currency | character varying(3) | | |
paid_date | timestamp without time zone | | |
Indexes:
"paid_date_t_paid_date_idx" btree (paid_date)
I am running below select query and joining these tables based on multiple join keys:
SELECT
paid_date
FROM exchange_rate erd
JOIN paid_date_t sspd
ON sspd.paid_date >= erd.valid_from AND sspd.paid_date < erd.valid_until
AND erd.currency = sspd.currency
WHERE sspd.currency != 'USD'
However, the performance of the query is inefficient and takes hours to execute. The query plan below shows that it using a nested loop join.
Nested Loop (cost=0.28..44498192.71 rows=701389198 width=40)
-> Seq Scan on paid_date_t sspd (cost=0.00..183612.84 rows=2557615 width=24)
Filter: ((currency)::text <> 'USD'::text)
-> Index Scan using exchange_rate_valid_from_until_currency_gist_idx on exchange_rate erd (cost=0.28..16.53 rows=80 width=36)
Index Cond: (currency = (sspd.currency)::text)
Filter: ((sspd.paid_date >= valid_from) AND (sspd.paid_date < valid_until))
I have worked with different indexing methods but got the same result. I know that <= and >= operators are not supporting merge or hash joins.
Any ideas are appreciated.
You should create a smaller table with just a sample of the rows from paid_date_t in it. It is hard to optimize a query if it takes a very long time each time you try to test it.
Your btree index has the column tested for equality as the 2nd column, which is certainly less efficient. The better btree index for this query (as it is currently written) would be something like (currency, valid_from, valid_until).
For a gist index, you really want it to be on the time range, not on the separate end points of the range. You could either convert the table to hold a range type, or build a functional index to convert them on the fly (and then rewrite the query to use the same expression). This is complicated by the fact that your tables have different types due to the different handling of time zones. The index would look like:
create index on exchange_rate using gist (tstzrange(valid_from,valid_until), currency);
and then the ON condition would look like:
ON sspd.paid_date::timestamptz <# tstzrange(erd.valid_from, erd.valid_until)
AND erd.currency = sspd.currency
It might be faster to have the order of the columns in the gist index be reversed from what I show, you should try it both ways on your own data and see.
PostgresSQL 11.2
Settings:
shared_buffers = 1024MB
effective_cache_size = 2048MB
maintenance_work_mem = 320MB
checkpoint_completion_target = 0.5
wal_buffers = 3932kB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 64MB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
I've got a table with about 40M rows.
The query I'm doing on it(more fields exist in the query, it's the where clauses that count):
select id,name from my_table where
action_performed = true AND
should_still_perform_action = false AND
action_performed_at <= '2021-09-05 00:00:00.000'
LIMIT 100;
The date is just something I picked for this example.
The point is to get items that need to be processed. The client retrieving this data would then use the metadata to find a file and upload it to a cloud provider. This can take some time.
The timestamp condition is really only there to say "only process entries older than today" or the given timestamp, in general. The order in which they are returned is of no practical importance, since the goal is to perform processing on any entry that has not yet been processed. The LIMIT was introduced to stop the application doing so from hanging, because of the network activity.
Table definition(redacted):
Table "public.my_table"
action_performed_at | timestamp without time zone | | | now()
should_still_perform_action | boolean | | not null | true
action_performed | boolean | | not null | false
Indexes:
"index001" btree (action_performed_at, should_still_perform_action, action_performed) WHERE should_still_perform_action = false AND action_performed = true
"index002" btree (action_performed, should_still_perform_action, action_performed_at DESC) WHERE should_still_perform_action = false AND action_performed = true
These are all indexes added over time, all worked at the start, but are no longer being used now.
Re-indexing also does not seem to work, only dropping and re-creating them works for a while.
While the table hold 40M rows, the amount of rows matching these conditions is roughly around 100K.
The query plan looks like this:
QUERY PLAN
------------------------------------------------------------------------------------
Limit (cost=0.00..707.80 rows=100 width=3595) (actual time=18520.627..100644.933 rows=100 loops=1)
Buffers: shared hit=0 read=1392361 dirtied=26 written=26
-> Seq Scan on my_table (cost=0.00..4164264.45 rows=5883377 width=3595) (actual time=18520.624..100644.073 rows=100 loops=1)
Filter: (action_performed AND (NOT should_still_perform_action) AND (action_performed_at <= '2021-09-05 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 19846606
Buffers: shared hit=0 read=1392361 dirtied=26 written=26
Planning Time: 63.667ms
Execution Time: 100645.548 ms
(10 rows)
Using the query found here: https://github.com/ioguix/pgsql-bloat-estimation/blob/master/btree/btree_bloat-superuser.sql
This is the result:
current_database | schemaname | tblname | idxname | real_size | extra_size | extra_pct | fillfactor | bloat_size | bloat_pct | is_na
------------------+------------+------------------+---------------------------------------+-------------+-------------+------------------+------------+-------------+------------------+-------
mine | public | my_table | index001 | 343244800 | 341598208 | 99.5202863961814 | 90 | 341426176 | 99.4701670644391 | f
mine | public | my_table | index002 | 3290316800 | 2338521088 | 71.0728245985311 | 90 | 2231902208 | 67.832441180132 | f
And I'm looking for a way to do this better. Sure, I could drop and recreate the index after time I see it slow down, but that's not exactly a good way of doing things.
Changing LIMIT to FETCH changes nothing.
I'm wondering if I can improve this without changing SELECT to FETCH, which I've never used before and I'm not even sure the client can handle.
What should I do here?
EDIT:
After an analyze:
QUERY PLAN
------------------------------------------------------------------------------------
Limit (cost=0.00..690.14 rows=1000 width=3591) (actual time=0.044..5840.228 rows=1000 loops=1)
Buffers: shared hit=3 read=81426 dirtied=18 written=18
-> Seq Scan on my_table (cost=0.00..4163978.60 rows=6033500 width=3591) (actual time=0.034..5839.599 rows=100 loops=1)
Filter: (action_performed AND (NOT should_still_perform_action) AND (action_performed_at <= '2021-09-05 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 953640
Buffers: shared hit=3 read=81426 dirtied=18 written=18
Planning Time: 63.667ms
Execution Time: 100645.548 ms
(10 rows)
The estimate (cost=0.00..4164264.45 rows=5883377 width=3595) shows the planner expects over 5M records to match the criteria. It is significantly different from the expected 100K you mention.
In cases like this `ANALYZE public.my_table;' usually helps. It refreshes statistics for the table data.
Your main problem seems to be index bloat, which is caused by more DELETEs or UPDATEs than autovacuum can clean up. Solve that problem, and you should be fine.
Tune autovacuum to be as fast as possible:
ALTER TABLE my_table SET (autovacuum_vacuum_cost_delay = 0);
Also, give autovacuum enough memory by setting maintenance_work_mem to a high value up to 1GB.
Then rebuild the indexes so that they are no longer bloated. If pgstattuple tells you that the table is bloated too, VACUUM (FULL) that table instead.
Make sure that you don't have long running database transactions most of the time.
By the way, this is the perfect index for this query:
CREATE INDEX ON my_table (action_performed_at)
WHERE action_performed AND NOT should_still_perform_action;
Entire time is taking to scan the table as per the plan. Seq scan is happening, even index is available on the required columns. It seems rows return by this query is quite high, not only 100k rows.
If you check the below condition in the plan, around 20M rows are removed by all 3 filter used in where clause in the query.
Rows Removed by Filter: 19846606
Please check
Why index isn't picking by reviewing the cardinality of the columns,
how many exact rows return by the query and When this table last
analyze.
is Autovaccuum enable in this database? when the last autovaccuum run for this table ?
Because of the statistics that are collected behind the index, the distribution histogram present detailed values only for the first column of the index key.
If this first column has only 2 values the accuracy is inconsistent and the optimizer will create a bad execution plan.
To bypass this trouble, you must place the column action_performed_at as the first column of the index key.
Another point is that you do not need to have column stored in an index with a single value. When you create an index with a WHERE clause that's rely on MyColum = A_Single_Value, you can ignore this column into the index key.
Finally you can use the INCLUDE clause, that MS SQL Server invented 16 years ago and arrived in PostGreSQL, to add some more columns that do not participate in any seek, but is necessary for the SELECT. This will use only the index and do not use a two phase access SEEK for index and SCAN for the table.
So I will try an index like this one :
CREATE INDEX SQLpro__B6B13FC3_6F90_4EEC_BA61_CA6C96C7958A__20210914
ON my_table (action_performed_at)
INCLUDE (id, name)
WHERE action_performed = true AND
should_still_perform_action = true;
I'm working on implementing a fast text search in PostgreSQL 12.6 (Ubuntu 20.04.2 VBox) with custom sorting, and I'm using pg_trgm along with GiST (btree_gist) index for sorted output. The idea is to return top 5 matching artists that have the highest number of plays. The index is created like this:
create index artist_name_gist_idx on artist
using gist ("name" gist_trgm_ops, total_play_count) where active = true;
"name" here is of type varchar(255) and total_play_count is bigint, no nulls allowed.
When I query the table like this:
select id, name, total_play_count
from artist
where name ilike '%the do%' and active = true
order by total_play_count <-> 40312
limit 5;
I get the correct result:
id | name | total_play_count
--------+-------------------------+------------------
1757 | The Doors | 1863
733226 | Damsel in the Dollhouse | 1095
9758 | The Doobie Brothers | 1036
822805 | The Doubleclicks | 580
7236 | Slaughter and the Dogs | 258
I would get the same result if I replace total_play_count <-> 40312 with simple total_play_count desc, but then I get the additional sort operation that I want to avoid. Number 40312 here is the current maximum value of this column, and table itself contains 1612297 rows in total.
However, since total_play_count is of type bigint, I wanted to make this query more general (and faster) and use the maximum value for bigint, so I don't have to query for the max value every time. But when I update the ORDER BY clause with total_play_count <-> 9223372036854775807, I get the following result:
id | name | total_play_count
---------+-------------------------+------------------
1757 | The Doors | 1863
822805 | The Doubleclicks | 580
9758 | The Doobie Brothers | 1036
733226 | Damsel in the Dollhouse | 1095
1380763 | Bruce Bawss The Don | 10
The ordering here is broken, and it's even worse when I try the same approach on another table that has a lot more rows. There are no negative or overly large values, so overflow shouldn't be possible. Results of explain are almost identical:
Limit (cost=0.41..6.13 rows=5 width=34)
-> Index Scan using artist_name_gist_idx on artist (cost=0.41..184.44 rows=161 width=34)
Index Cond: ((name)::text ~~* '%the do%'::text)
Order By: (total_play_count <-> '9223372036854775807'::bigint)
What could be the issue here? Is this a bug with btree_gist, or am I missing something? I could settle for querying for the max value, but it worries me that there is a threshold that might be reached eventually and break the search, which would be a shame since I'm quite happy with the performance.
Update:
I've tried using regular integer type instead of bigint, and then query with it's upper bound total_play_count <-> 2147483647. It seems that there is no such problem with it. Perhaps using bigint in the first place was somewhat optimistic, but I'll keep the question open in case anyone has an answer or a workaround.
I could use some help optimizing a query that compares rows in a single table with millions of entries. Here's the table's definition:
CREATE TABLE IF NOT EXISTS data.row_check (
id uuid NOT NULL DEFAULT NULL,
version int8 NOT NULL DEFAULT NULL,
row_hash int8 NOT NULL DEFAULT NULL,
table_name text NOT NULL DEFAULT NULL,
CONSTRAINT row_check_pkey
PRIMARY KEY (id, version)
);
I'm reworking our push code and have a test bed with millions of records across about 20 tables. I run my tests, get the row counts, and can spot when some of my insert code has changed. The next step is to checksum each row, and then compare the rows for differences between versions of my code. Something like this:
-- Run my test of "version 0" of the push code, the base code I'm refactoring.
-- Insert the ID and checksum for each pushed row.
INSERT INTO row_check (id,version,row_hash,table_name)
SELECT id, 0, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_check_pkey DO UPDATE SET
row_hash = EXCLUDED.row_hash,
table_name = EXCLUDED.table_name;
truncate table record_changes_log;
-- Run my test of "version 1" of the push code, the new code I'm validating.
-- Insert the ID and checksum for each pushed row.
INSERT INTO row_check (id,version,row_hash,table_name)
SELECT id, 1, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_check_pkey DO UPDATE SET
row_hash = EXCLUDED.row_hash,
table_name = EXCLUDED.table_name;
That gets two rows in row_check for every row in record_changes_log, or any other table I'm checking. For the two runs of record_changes_log, I end up with more than 8.6M rows in row_check. They look like this:
id version row_hash table_name
e6218751-ab78-4942-9734-f017839703f6 0 -142492569 record_changes_log
6c0a4111-2f52-4b8b-bfb6-e608087ea9c1 0 -1917959999 record_changes_log
7fac6424-9469-4d98-b887-cd04fee5377d 0 -323725113 record_changes_log
1935590c-8d22-4baf-85ba-00b563022983 0 -1428730186 record_changes_log
2e5488b6-5b97-4755-8a46-6a46317c1ae2 0 -1631086027 record_changes_log
7a645ffd-31c5-4000-ab66-a565e6dad7e0 0 1857654119 record_changes_log
I asked yesterday for some help on the comparison query, and it lead to this:
select v0.table_name,
v0.id,
v0.row_hash as v0,
v1.row_hash as v1
from row_check v0
join row_check v1 on v0.id = v1.id and
v0.version = 0 and
v1.version = 1 and
v0.row_hash <> v1.row_hash
That works, but now I'm hoping to optimize it a bit. As an experiment, I clustered the data on version and then built a BRIN index, like this:
drop index if exists row_check_version_btree;
create index row_check_version_btree
on row_check
using btree(version);
cluster row_check using row_check_version_btree;
drop index row_check_version_btree; -- Eh? I want to see how the BRIN performs.
drop index if exists row_check_version_brin;
create index row_check_version_brin
on row_check
using brin(row_hash);
vacuum analyze row_check;
I ran the query through explain analyze and get this:
Merge Join (cost=1.12..559750.04 rows=4437567 width=51) (actual time=1511.987..14884.045 rows=10 loops=1)
Output: v0.table_name, v0.id, v0.row_hash, v1.row_hash
Inner Unique: true
Merge Cond: (v0.id = v1.id)
Join Filter: (v0.row_hash <> v1.row_hash)
Rows Removed by Join Filter: 4318290
Buffers: shared hit=8679005 read=42511
-> Index Scan using row_check_pkey on ascendco.row_check v0 (cost=0.56..239156.79 rows=4252416 width=43) (actual time=0.032..5548.180 rows=4318300 loops=1)
Output: v0.id, v0.version, v0.row_hash, v0.table_name
Index Cond: (v0.version = 0)
Buffers: shared hit=4360752
-> Index Scan using row_check_pkey on ascendco.row_check v1 (cost=0.56..240475.33 rows=4384270 width=24) (actual time=0.031..6070.790 rows=4318300 loops=1)
Output: v1.id, v1.version, v1.row_hash, v1.table_name
Index Cond: (v1.version = 1)
Buffers: shared hit=4318253 read=42511
Planning Time: 1.073 ms
Execution Time: 14884.121 ms
...which I did not really get the right idea from...so I ran it again to JSON and fed the results into this wonderful plan visualizer:
http://tatiyants.com/pev/#/plans
The tips there are right, the top node estimate is bad. The result is 10 rows, the estimate is for about 443,757 rows.
I'm hoping to learn more about optimizing this kind of thing, and this query seems like a good opportunity. I have a lot of notions about what might help:
-- CREATE STATISTICS?
-- Rework the query to move the where comparison?
-- Use a better index? I did try a GIN index and a straight B-tree on version, but neither was superior.
-- Rework the row_check format to move the two hashes into the same row instead of splitting them over two rows, compare on insert/update, flag non-matches, and add a partial index for the non-matching values.
Granted, it's funny to even try to index something where there are only two values (0 and 1 in the case above), so there's that. In fact, is there any sort of clever trick for Booleans? I'll always be comparing two versions, so "old" and "new", which I can express however makes life best. I understand that Postgres only has bitmap indexes internally at search/merge (?) time and that it does not have a bitmap type index. Would there be some kind of INTERSECT that might help? I don't know how Postgres implements set math operators internally.
Thanks for any suggestions on how to rethink this data or the query to make it faster for comparisons involving millions, or tens of millions, of rows.
I'm going to add an answer to my own question, but am still interested in what anyone else has to say. In the process of writing out my original question, I realized that maybe a redesign is in order. This hinges on my plan to only ever compare two versions at a time. That's a good solution here, but there are other cases where it wouldn't work. Anyway, here's a slightly different table design that folds the two results into a single row:
DROP TABLE IF EXISTS data.row_compare;
CREATE TABLE IF NOT EXISTS data.row_compare (
id uuid NOT NULL DEFAULT NULL,
hash_1 int8, -- Want NULL to defer calculating hash comparison until after both hashes are entered.
hash_2 int8, -- Ditto
hashes_match boolean, -- Likewise
table_name text NOT NULL DEFAULT NULL,
CONSTRAINT row_compare_pkey
PRIMARY KEY (id)
);
The following expression index should, hopefully, be very small as I shouldn't have any non-matching entries:
CREATE INDEX row_compare_fail ON row_compare (hashes_match)
WHERE hashes_match = false;
The trigger below does the column calculation, once hash_1 and hash_2 are both provided:
-- Run this as a BEFORE INSERT or UPDATE ROW trigger.
CREATE OR REPLACE FUNCTION data.on_upsert_row_compare()
RETURNS trigger AS
$BODY$
BEGIN
IF NEW.hash_1 = NULL OR
NEW.hash_2 = NULL THEN
RETURN NEW; -- Don't do the comparison, hash_1 hasn't been populated yet.
ELSE-- Do the comparison. The point of this is to avoid constantly thrashing the expression index.
NEW.hashes_match := NEW.hash_1 = NEW.hash_2;
RETURN NEW; -- important!
END IF;
END;
$BODY$
LANGUAGE plpgsql;
This now adds 4.3M rows instead of 8.6M rows:
-- Add the first set of results and build out the row_compare records.
INSERT INTO row_compare (id,hash_1,table_name)
SELECT id, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_compare_pkey DO UPDATE SET
hash_1 = EXCLUDED.hash_1,
table_name = EXCLUDED.table_name;
-- I'll truncate the record_changes_log and push my sample data again here.
-- Add the second set of results and update the row compare records.
-- This time, the hash is going into the hash_2 field for comparison
INSERT INTO row_compare (id,hash_2,table_name)
SELECT id, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_compare_pkey DO UPDATE SET
hash_2 = EXCLUDED.hash_2,
table_name = EXCLUDED.table_name;
And now the results are simple to find:
select * from row_compare where hashes_match = false;
This changes the query time from around 17 seconds to around 24 milliseconds.
I'm beating my head on this since yesterday, and I don't understanf what's happening:
I am populating a dimensional schema for a datawarehousing project, using Pentaho Kettle to perform a "dimension lookup/update", which basically looks up for existing rows in a dimension table, inserting the ones which do not exist and returning the technical key.
The dimension table itself is very simple:
CREATE TABLE dim_loan
(
_tech_id INTEGER NOT NULL,
loan_id INTEGER,
type TEXT,
interest_rate_type TEXT,
_dim_project_id integer,
_validity_from date,
_validity_to date,
_version integer,
PRIMARY KEY (_tech_id)
);
CREATE INDEX dim_loan_pk_idx ON dim_loan USING btree (_tech_id);
CREATE INDEX dim_loan_compound_idx ON dim_loan USING btree (loan_id, _dim_project_id, _validity_from, _validity_to);
The table should contain, at the end of the process, around 650k rows. The transformations starts fast(ish), at around 1500 rows/sec.
The performance drops steadily reaching 50 rows/sec by the time the table has around 50k rows.
The queries that Kettle does look like this:
SELECT _tech_id, _version, "type" AS "Loan Type", interest_rate_type AS int_rate, _validity_from, _validity_to FROM "public".dim_loan WHERE loan_id = $1 AND _dim_project_id = $2 AND $3 >= _validity_from AND $4 < _validity_to
The query planner estimates an execution time of 0.1 msecs:
"Index Scan using dim_loan_compound_idx on dim_loan (cost=0.42..7.97 rows=1 width=42) (actual time=0.043..0.043 rows=0 loops=1)"
" Index Cond: ((loan_id = 1) AND (_dim_project_id = 2) AND ('2016-01-01'::date >= _validity_from) AND ('2016-01-01'::date < _validity_to))"
"Total runtime: 0.078 ms"
Of course real execution times are much different, around 10ms, which is unacceptable. Enabling slow query log with auto_explain I see with increased frequency entries like this:
Seq Scan on dim_loan (cost=0.00..2354.21 rows=12 width=52)
Filter: (($3 >= _validity_from) AND ($4 < _validity_to) AND (_dim_project_id = $2) AND ((loan_id)::double precision = $1))
< 2016-12-18 21:30:19.859 CET >LOG: duration: 14.260 ms plan:
Query Text: SELECT _tech_id, _version, "type" AS "Loan Type", interest_rate_type AS int_rate, _validity_from, _validity_to FROM "public".dim_loan WHERE loan_id = $1 AND _dim_project_id = $2 AND $3 >= _validity_from
AND $4 < _validity_to
Which don't tell the whole story anyway as it's not only these queries that run slow, but all of them.
Of course I tried to tweak the memory parameters up to silly amounts with no real difference in performance, I also tried the latest 9.6, which exhibited the same behavior as 9.3, which is what I'm using.
The same transformation, on a MySQL database with the same indexes, runs happily at 5000 rows/sec from start to finish. I really want to use PG and I'm sure that it's something trivial, but what!?
Maybe something with the jdbc driver? I verified that it does use a single connection all the time, so it's not even a connection overhead issue...
Just found out that the cause is indeed loan id being cast to double, which of course rendered the index useless! The reason is a wrong assumption made by Kettle on the metadata of this column, which comes from an excel file.
Now the performance is on par with MySQL! Happy days