I have created the below example and do not understand why the planner does not use index i2 for the query. As can be seen in pg_stats, it understands that column uniqueIds contains unique values. it also understands that column fourOtherIds contains only 4 different values. Shouldn't a search of index i2 then be by far the fastest way? Looking for uniqueIds in only four different index leaves of fourOtherIds? What is wrong with my understanding of how an index works? Why does it think using i1 makes more sense here, even though it has to filter out 333.333 rows? In my understanding it should use i2 to find the one row (or few rows, as there is no unique constraint) that has uniqueIds 4000 first and then apply where fourIds = 1 as a filter.
create table t (fourIds int, uniqueIds int,fourOtherIds int);
insert into t ( select 1,*,5 from generate_series(1 ,1000000));
insert into t ( select 2,*,6 from generate_series(1000001,2000000));
insert into t ( select 3,*,7 from generate_series(2000001,3000000));
insert into t ( select 4,*,8 from generate_series(3000001,4000000));
create index i1 on t (fourIds);
create index i2 on t (fourOtherIds,uniqueIds);
analyze t;
select n_distinct,attname from pg_stats where tablename = 't';
/*
n_distinct|attname |
----------+------------+
4.0|fourids |
-1.0|uniqueids |
4.0|fourotherids|
*/
explain analyze select * from t where fourIds = 1 and uniqueIds = 4000;
/*
QUERY PLAN |
--------------------------------------------------------------------------------------------------------------------------+
Gather (cost=1000.43..22599.09 rows=1 width=12) (actual time=0.667..46.818 rows=1 loops=1) |
Workers Planned: 2 |
Workers Launched: 2 |
-> Parallel Index Scan using i1 on t (cost=0.43..21598.99 rows=1 width=12) (actual time=25.227..39.852 rows=0 loops=3)|
Index Cond: (fourids = 1) |
Filter: (uniqueids = 4000) |
Rows Removed by Filter: 333333 |
Planning Time: 0.107 ms |
Execution Time: 46.859 ms |
*/
Not every conceivable optimization has been implemented. You are looking for a variant of an index skip scan AKA a loose index scan. PostgreSQL does not automatically implement those (yet--people were working on it but I don't know if they still are. Also, I think I've read that one of the 3rd party extensions/forks, citus maybe, has implemented it). You can emulate one yourself using a recursive CTE, but that would be quite annoying to do.
Related
Postgres is using a Nested Loop Join algorithm when I use a non equi join condition in my update query. I understand that the Nested Loop Join can be very costly as the right relation is scanned once for every row found in the left relation as per
[https://www.postgresql.org/docs/8.3/planner-optimizer.html]
The update query and the execution plan is below.
Query
explain analyze
UPDATE target_tbl tgt
set descr = stage.descr,
prod_name = stage.prod_name,
item_name = stage.item_name,
url = stage.url,
col1_name = stage.col1_name,
col2_name = stage.col2_name,
col3_name = stage.col3_name,
col4_name = stage.col4_name,
col5_name = stage.col5_name,
col6_name = stage.col6_name,
col7_name = stage.col7_name,
col8_name = stage.col8_name,
flag = stage.flag
from tbl1 stage
where tgt.col1 = stage.col1
and tgt.col2 = stage.col2
and coalesce(tgt.col3, 'col3'::text) = coalesce(stage.col3, 'col3'::text)
and coalesce(tgt.col4, 'col4'::text) = coalesce(stage.col4, 'col4'::text)
and stage.row_number::int >= 1::int
and stage.row_number::int < 50001::int;
Execution Plan
Update on target_tbl tgt (cost=0.56..3557.91 rows=1 width=813) (actual time=346153.460..346153.460 rows=0 loops=1)
-> Nested Loop (cost=0.56..3557.91 rows=1 width=813) (actual time=4.326..163876.029 rows=50000 loops=1)
-> Seq Scan on tbl1 stage (cost=0.00..2680.96 rows=102 width=759) (actual time=3.060..2588.745 rows=50000 loops=1)
Filter: (((row_number)::integer >= 1) AND ((row_number)::integer < 50001))
-> Index Scan using tbl_idx on target_tbl tgt (cost=0.56..8.59 rows=1 width=134) (actual time=3.152..3.212 rows=1 loops=50000)
Index Cond: ((col1 = stage.col1) AND (col2 = stage.col2) AND (COALESCE(col3, 'col3'::text) = COALESCE(stage.col3, 'col3'::text)) AND (COALESCE(col4, 'col4'::text) = COALESCE(stage.col4, 'col4'::text)))
Planning time: 17.700 ms
Execution time: 346157.168 ms
Is there any way to avoid the nested loop join during the execution of the above query?
Or is there a way that can help me to reduce the cost of the the nested loop scan, currently it takes 6-7 minutes to update just 50000 records?
PostgreSQL can choose a different join strategy in that case. The reason why it doesn't is the gross mis-estimate in the sequential scan: 102 instead of 50000.
Fix that problem, and things will get better:
ANALYZE tbl1;
If that is not enough, collect more detailed statistics:
ALTER TABLE tbl1 ALTER row_number SET STATISTICS 1000;
ANALYZE tbl1;
All this assumes that row_number is an integer and the type cast is redundant. If you made the mistake to use a different data type, an index is your only hope:
CREATE INDEX ON tbl1 ((row_number::integer));
ANALYZE tbl1;
I understand that the Nested Loop Join can be very costly as the right relation is scanned once for every row found in the left relation
But the "right relation" here is an index scan, not a scan of the full table.
You can get it to stop using the index by changing the leading column of the join condition to something like where tgt.col1+0 = stage.col1 .... Upon doing that, it will probably change to a hash join or a merge join, but you will have to try it and see if it does. Also, the new plan may not actually be faster. (And fixing the estimation problem would be preferable, if that works)
Or is there a way that can help me to reduce the cost of the the
nested loop scan, currently it takes 6-7 minutes to update just 50000
records?
Your plan shows that over half the time is spent on the update itself, so probably reducing the cost of just the nested loop scan can have only a muted impact on the overall time. Do you have a lot of indexes on the table? The maintenance of those indexes might be a major bottleneck.
I have an sql query on postgresql 9.5, but it takes too long time. And I run the explain query:
DELETE FROM source v1
WHERE id < (SELECT MAX(id)
FROM source v2
WHERE v2.ent_id = v1.ent_id
AND v2.name = v1.name
);
And ex plain is
Delete on source v1 (cost=0.00..1764410287608.21 rows=2891175 width=6)');
-> Seq Scan on source v1 (cost=0.00..1764410287608.21 rows=2891175 width=6)');
Filter: (id < (SubPlan 2))');
SubPlan 2');
-> Result (cost=203424.76..203424.77 rows=1 width=0)');
InitPlan 1 (returns $2)');
-> Limit (cost=0.43..203424.76 rows=1 width=8)');
-> Index Scan Backward using source_id_ix on source v2 (cost=0.43..813697.74 rows=4 width=8)');
Index Cond: (id IS NOT NULL)');
Filter: (((ent_id)::text = (v1.ent_id)::text) AND ((name)::text = (v1.name)::text))');
My table has about 8.000.000 records. And I could not get the result for days. And I could not calculate how many times will take? is there any way for a new solution?
There is no really good way to predict execution time.
As a very rough rule of thumb, you can compare a cost of 1 to the time of reading one 8 KB page from disk during a sequential scan, but that will often be off by more than an order of magnitude.
To solve the underlying problem, try
DELETE FROM source AS v1
WHERE EXISTS (SELECT 1
FROM source AS v2
WHERE (v1.ent_id, v1.name) = (v2.ent_id, v2.name)
AND v2.id > v1.id);
The problem with your query is that it has to execute an expensive subselect for every row found, while mine can perform a semijoin. Look at my query's execution plan.
We have a table with 4 million records, and for a particular frequently used use-case we are only interested in records with a particular salesforce userType of 'Standard' which are only about 10,000 out of 4 million. The other usertype's that could exist are 'PowerPartner', 'CSPLitePortal', 'CustomerSuccess', 'PowerCustomerSuccess' and 'CsnOnly'.
So for this use case I thought creating a partial index would be better, as per the documentation.
So I am planning to create this partial index to speed up queries for records with a usertype of 'Standard' and prevent the request from the web from getting timed out:
CREATE INDEX user_type_idx ON user_table(userType)
WHERE userType NOT IN
('PowerPartner', 'CSPLitePortal', 'CustomerSuccess', 'PowerCustomerSuccess', 'CsnOnly');
The lookup query will be
select * from user_table where userType='Standard';
Could you please confirm if this is the right way to create the partial index? It would of great help.
Postgres can use that but it does so in a way that is (slightly) less efficient than an index specifying where user_type = 'Standard'.
I created a small test table with 4 million rows, 10.000 of them having the user_type 'Standard'. The other values were randomly distributed using the following script:
create table user_table
(
id serial primary key,
some_date date not null,
user_type text not null,
some_ts timestamp not null,
some_number integer not null,
some_data text,
some_flag boolean
);
insert into user_table (some_date, user_type, some_ts, some_number, some_data, some_flag)
select current_date,
case (random() * 4 + 1)::int
when 1 then 'PowerPartner'
when 2 then 'CSPLitePortal'
when 3 then 'CustomerSuccess'
when 4 then 'PowerCustomerSuccess'
when 5 then 'CsnOnly'
end,
clock_timestamp(),
42,
rpad(md5(random()::text), (random() * 200 + 1)::int, md5(random()::text)),
(random() + 1)::int = 1
from generate_series(1,4e6 - 10000) as t(i)
union all
select current_date,
'Standard',
clock_timestamp(),
42,
rpad(md5(random()::text), (random() * 200 + 1)::int, md5(random()::text)),
(random() + 1)::int = 1
from generate_series(1,10000) as t(i);
(I create tables that have more than just a few columns as the planner's choices are also driven by the size and width of the tables)
The first test using the index with NOT IN:
create index ix_not_in on user_table(user_type)
where user_type not in ('PowerPartner', 'CSPLitePortal', 'CustomerSuccess', 'PowerCustomerSuccess', 'CsnOnly');
explain (analyze true, verbose true, buffers true)
select *
from user_table
where user_type = 'Standard'
Results in:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on stuff.user_table (cost=139.68..14631.83 rows=11598 width=139) (actual time=1.035..2.171 rows=10000 loops=1)
Output: id, some_date, user_type, some_ts, some_number, some_data, some_flag
Recheck Cond: (user_table.user_type = 'Standard'::text)
Buffers: shared hit=262
-> Bitmap Index Scan on ix_not_in (cost=0.00..136.79 rows=11598 width=0) (actual time=1.007..1.007 rows=10000 loops=1)
Index Cond: (user_table.user_type = 'Standard'::text)
Buffers: shared hit=40
Total runtime: 2.506 ms
(The above is a typical execution time after I ran the statement about 10 times to eliminate caching issues)
As you can see the planner uses a Bitmap Index Scan which is a "lossy" scan that needs an extra step to filter out false positives.
When using the following index:
create index ix_standard on user_table(id)
where user_type = 'Standard';
This results in the following plan:
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Index Scan using ix_standard on stuff.user_table (cost=0.29..443.16 rows=10267 width=139) (actual time=0.011..1.498 rows=10000 loops=1)
Output: id, some_date, user_type, some_ts, some_number, some_data, some_flag
Buffers: shared hit=313
Total runtime: 1.815 ms
Conclusion:
Your index is used but an index on only the type that you are interested in is a bit more efficient.
The runtime is not that much different. I executed each explain about 10 times, and the average for the ix_standard index was slightly below 2ms and the average of the ix_not_in index was slightly above 2ms - so not a real performance difference.
But in general the Index Scan will scale better with increasing table sizes than the Bitmap Index Scan will do. This is basically due to the "Recheck Condition" - especially if not enough work_mem is available to keep the bitmap in memory (for larger tables).
For the index to be used, the WHERE condition must be used in the query as you wrote it.
PostgreSQL has some ability to make deductions, but it won't be able to infer that userType = 'Standard' is equivalent to the condition in the index.
Use EXPLAIN to find out if your index can be used.
I have a table with 10s of millions of rows. Various complex filtering queries produce row sets to support an application. These row sets are of of arbitrary size from a single row up to and including the full table. For domain-specific reasons, however, they always maintain high levels of contiguity along a particular key.
I need to pass these row sets bidirectionally between the database and the application, and it would be nice to compress this somehow. Many of you are probably familiar with UNIX cut, which takes a field specification like so: cut -f 2-6,7,9-21 and returns the corresponding columns. I am currently using a slightly limited version of the cut field specification (e.g. no 17-) to represent the row sets. So, for instance 24-923817,2827711-8471362,99188271 indicates a unique set of 6567445 rows while occupying 34 bytes.
I have already written the following procedures to convert these to SQL WHERE filters using the BETWEEN syntax
CREATE OR REPLACE FUNCTION cut_string_to_sql_filter( TEXT, TEXT ) RETURNS TEXT AS $$
SELECT
CASE $1
WHEN '' THEN 'FALSE'
ELSE
(SELECT
'(' || STRING_AGG( REGEXP_REPLACE( REGEXP_REPLACE( str, '(\d+)-(\d+)', QUOTE_IDENT( $2 ) || ' BETWEEN \1 AND \2' ), '^(\d+)$', QUOTE_IDENT( $2 ) || '=\1' ), ' OR ' ) || ')' AS sql
FROM
REGEXP_SPLIT_TO_TABLE( $1, ',' ) AS t(str))
END;
$$ LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;
The first parameter is the row set specification and the second parameter is the key field name for the table. For the example above, SELECT cut_string_to_sql_filter( '24-923817,2827711-8471362,99188271', 'some_key' ) returns:
(some_key BETWEEN 24 AND 923817 OR some_key BETWEEN 2827711 AND 8471362 OR some_key=99188271)
The problem with this is that currently any query that makes use of such row set specifications must use dynamic SQL, because I cannot think of a way to use custom operators or any other syntactic features to embed this effect in a plain SQL query.
I have also written a set-returning function for the row specifications:
CREATE OR REPLACE FUNCTION cut_string_to_set( TEXT ) RETURNS SETOF INTEGER AS $$
DECLARE
_i TEXT;
_j TEXT;
_pos INTEGER;
_start INTEGER;
_end INTEGER;
BEGIN
IF $1 <> '' THEN
FOR _i IN SELECT REGEXP_SPLIT_TO_TABLE( $1, ',' ) LOOP
_pos := POSITION( '-' IN _i );
IF _pos > 0 THEN
_start := SUBSTRING( _i FROM 1 FOR _pos - 1 )::INTEGER;
_end := SUBSTRING( _i FROM _pos + 1 )::INTEGER;
FOR _j IN _start.._end LOOP
RETURN NEXT _j;
END LOOP;
ELSE
RETURN NEXT _i;
END IF;
END LOOP;
END IF;
END
$$ LANGUAGE PLPGSQL IMMUTABLE STRICT PARALLEL SAFE;
This works in plain SQL with WHERE some_key IN (SELECT cut_string_to_set(...)). It is, of course, comparatively inefficient in unpacking what is best expressed to the planner as a set of ranges, produces nightmarish and verbose query plans, and may or may not prevent the planner from using an index when it otherwise could and should.
Can anybody offer any solutions to the above conundrum for packaging this, potentially as its own type, potentially with custom operators, to allow syntactically sane index-based filtering on a column without dynamic SQL in the broader involved query? Is this simply impossible?
Feel free to offer suggestions for improvements to the procedures if you see any opportunities. And thanks!
EDIT 1
Great answer below suggests using an array of range types. Unfortunately, the query planner does not seem willing to use indexes with such a query. Planner output below from run on a small test table.
Gather (cost=1000.00..34587.33 rows=38326 width=45) (actual time=0.395..112.334 rows=1018 loops=1)
Workers Planned: 6
Workers Launched: 6
-> Parallel Seq Scan on test (cost=0.00..29754.73 rows=6388 width=45) (actual time=91.525..107.354 rows=145 loops=7)
Filter: (test_ref <# ANY ('{"[24,28)","[29,51)","[999,1991)"}'::int4range[]))
Rows Removed by Filter: 366695
Planning time: 0.214 ms
Execution time: 116.779 ms
The CPU cost (notice 6 workers in parallel for over 100 ms on the small test table) is too high. I cannot see how any additional index could help here.
To contrast, here is the planner output using the BETWEEN filters.
Bitmap Heap Scan on test (cost=22.37..1860.39 rows=1031 width=45) (actual time=0.134..0.430 rows=1018 loops=1)
Recheck Cond: (((test_ref >= 24) AND (test_ref <= 27)) OR ((test_ref >= 29) AND (test_ref <= 50)) OR ((test_ref >= 999) AND (test_ref <= 1990)))
Heap Blocks: exact=10
-> BitmapOr (cost=22.37..22.37 rows=1031 width=0) (actual time=0.126..0.126 rows=0 loops=1)
-> Bitmap Index Scan on test_test_ref_index (cost=0.00..2.46 rows=3 width=0) (actual time=0.010..0.010 rows=4 loops=1)
Index Cond: ((test_ref >= 24) AND (test_ref <= 27))
-> Bitmap Index Scan on test_test_ref_index (cost=0.00..2.64 rows=21 width=0) (actual time=0.004..0.004 rows=22 loops=1)
Index Cond: ((test_ref >= 29) AND (test_ref <= 50))
-> Bitmap Index Scan on test_test_ref_index (cost=0.00..16.50 rows=1007 width=0) (actual time=0.111..0.111 rows=992 loops=1)
Index Cond: ((test_ref >= 999) AND (test_ref <= 1990))
Planning time: 0.389 ms
Execution time: 0.660 ms
END EDIT 1
EDIT 2
Answer below suggest using range index. The problem, as far as I understand this, is that I do not need to index a range type. All right, so maybe the key column is converted to a range for the operation, so I can apply a GIST index to it and the planner will use that.
CREATE INDEX test_test_ref_gist_index ON test USING GIST (test_ref);
ERROR: data type integer has no default operator class for access method "gist"
HINT: You must specify an operator class for the index or define a default operator class for the data type.
No surprise here. So let's convert the key column to a range and index that.
CREATE INDEX test_test_ref_gist_index ON test USING GIST (INT4RANGE( test_ref, test_ref ));
Whew, a 110 MB index. That's hefty. But does it work.
Gather (cost=1000.00..34587.33 rows=38326 width=45) (actual time=0.419..111.009 rows=1018 loops=1)
Workers Planned: 6
Workers Launched: 6
-> Parallel Seq Scan on test_mv (cost=0.00..29754.73 rows=6388 width=45) (actual time=90.229..105.866 rows=145 loops=7)
Filter: (test_ref <# ANY ('{"[24,28)","[29,51)","[999,1991)"}'::int4range[]))
Rows Removed by Filter: 366695
Planning time: 0.237 ms
Execution time: 114.795 ms
Nope. I'm not too surprised. I would expect this index to work for "contains" rather than "contained by" operations. I have no experience here though.
END EDIT 2
Pass an array of ranges:
select *
from t
where
k <# any (array[
'[24,923817]','[2827711,8471362]','[99188271,99188271]'
]::int4range[])
Check indexing for range types: https://www.postgresql.org/docs/current/static/rangetypes.html#RANGETYPES-INDEXING
In case a suitable range index is not possible do a join to the materialized ranges:
select *
from
t
inner join
(
select generate_series(lower(a),upper(a) - 1) as k
from unnest(array[
'[24,27]','[29,50]','[999,1990]'
]::int4range[]) a(a)
) s using (k)
It is possible to avoid joining all the range values. Compare to the lower and upper bounds of the range:
select *
from
t
cross join
(
select lower(a) as l, upper(a) - 1 as u
from unnest(array[
'[24,27]','[29,50]','[999,1990]'
]::int4range[]) a(a)
) s
where k between l and u
Simply impossible. Operators don't do that. They call functions. If they called a function here, that would function would have to use dynamic SQL.
To not use dynamic SQL, you'd have to hack apart the PostgreSQL lexer. PostgreSQL is a SQL database. Your syntax is not SQL. You can do two things,
Use SQL.
Compile SQL.
I prefer the first option where possible. If I need to make a DSL though, I don't do it in PostgreSQL. I do it in the app.
I am confused about what Redshift is doing when I run 2 seemingly similar queries. Neither should return a result (querying a profile that doesn't exist). Specifically:
SELECT * FROM profile WHERE id = 'id_that_doesnt_exist' and project_id = 1;
Execution time: 36.75s
versus
SELECT COUNT(*) FROM profile WHERE id = 'id_that_doesnt_exist' and project_id = 1;
Execution time: 0.2s
Given that the table is sorted by project_id then id I would have thought this is just a key lookup. The SELECT COUNT(*) ... returns 0 results in 0.2sec which is about what I would expect. The SELECT * ... returns 0 results in 37.75sec. That's a huge difference for the same result and I don't understand why?
If it helps schema as follows:
CREATE TABLE profile (
project_id integer not null,
id varchar(256) not null,
created timestamp not null,
/* ... approx 50 other columns here */
)
DISTKEY(id)
SORTKEY(project_id, id);
Explain from SELECT COUNT(*) ...
XN Aggregate (cost=435.70..435.70 rows=1 width=0)
-> XN Seq Scan on profile (cost=0.00..435.70 rows=1 width=0)
Filter: (((id)::text = 'id_that_doesnt_exist'::text) AND (project_id = 1))
Explain from SELECT * ...
XN Seq Scan on profile (cost=0.00..435.70 rows=1 width=7356)
Filter: (((id)::text = 'id_that_doesnt_exist'::text) AND (project_id = 1))
Why is the non count much slower? Surely Redshift knows the row doesn't exist?
The reason is: in many RDBMS's the answer on count(*) question usually come without actual data scan: just from index or table statistics. Redshift stores minimal and maximal value for a block that used to give exist or not exists answers for example like in describer case. In case requested value inside of min/max block boundaries the scan will be performed only on filtering fields data. In case requested value is lower or upper block boundaries the answer will be given much faster on basis of the stored statistics. In case of "select * " question RedShift actually scans all columns data as asked in query: "*" but filter only by columns in "where " clause.