In my Postgres 14.3 database on AWS RDS, I want to create an index without blocking other database operations. So I want to use the CONCURRENTLY option and I executed the following statement successfully.
CREATE INDEX CONCURRENTLY idx_test
ON public.ap_identifier USING btree (cluster_id);
But when checking the database with:
SELECT * FROM pg_indexes WHERE indexname = 'idx_test';
I only see: Index without CONCURRENTLY option
I am expecting the index is created with CONCURRENTLY option.
Is there any database switch to turn this feature on, or why does it seem to ignore CONCURRENTLY?
As has been commented, CONCURRENTLY is not a property of the index, but an instruction to create the index without blocking concurrent writes. The resulting index does not remember that option in any way. Read the chapter "Building Indexes Concurrently" in the manual.
Creating indexes on big tables can take a while. The system table
pg_stat_progress_create_index can be queried for progress reporting. While that is going on, CONCURRENTLY is still reflected in the command column.
To your consolation: once created, all indexes are "concurrent" anyway, in the sense that they are maintained automatically without blocking concurrent reads or writes (except for UNIQUE indexes that prevent duplicates.)
Related
I have two aws task where one could be writing to table another could run delete query at the same time both could be running parallel queries but playing with different rows. so the question is if I run delete from table where column_name=some condition ? then will postgres apply table level lock or row level lock. If table level lock gets applied then another task will not able to write into table.
PostgreSQL will only lock the rows you delete.
Concurrent INSERTs will not be affected.
There are many different lock modes. The table will be locked, but in a mode that still allows other INSERT, DELETE, and UPDATE operations to happen concurrently. The rows actually deleted will also be locked, in a more restrictive mode.
Looking here, it is clear that Oracle supports execution of DDL commands in parallel with scenarios clearly listed. I was wondering whether Postgres does indeed offer such functionality? I can find a lot of material on "parallel queries" for PostgreSQL but not so much when DDL is involved.
For example, can I execute multiple 'CREATE TABLE...AS SELECT' in parallel? And if not, how can I achieve such functionality? What happens if I have a temporary table (CREATE TEMP TABLE)? Do I need to configure something for locks?
From here:
Even when it is in general possible for parallel query plans to be generated, the planner will not generate them for a given query if any of the following are true:
The query writes any data or locks any database rows. If a query contains a data-modifying operation either at the top level or within
a CTE, no parallel plans for that query will be generated.
(emphasis mine).
Which seems to suggest that Postgres will not "parallelize" any query that modifies the database structure, under any circumstances.
Running multiple queries simultaneously in Postgres requires one connection per running query.
Those are generic DDL statements, they are index operations and partition operations that can be parallelized.
If you check the Notes section of the CREATE INDEX statement, you'll see that parallel index building is supported :
PostgreSQL can build indexes while leveraging multiple CPUs in order to process the table rows faster. This feature is known as parallel index build. For index methods that support building indexes in parallel (currently, only B-tree), maintenance_work_mem specifies the maximum amount of memory that can be used by each index build operation as a whole, regardless of how many worker processes were started. Generally, a cost model automatically determines how many worker processes should be requested, if any.
Update
I suspect the real question is about CREATE TABLE ... AS though.
This is essentially a CREATE TABLE followed by an INSERT .. SELECT. The CREATE TABLE part can't be parallelized and doesn't have to - it's essentially a metadata operation. The SELECT on the other hand, could be parallelized easily. INSERT is a bit harder, but it's a matter of implementation.
As a_horse_with_no_name explains in a comment to this question, parallelization for CREATE TABLE AS was added in PostgreSQL 11 :
Improvements to parallelism, including:
CREATE INDEX can now use parallel processing while building a B-tree index
Parallelization is now possible in CREATE TABLE ... AS, CREATE MATERIALIZED VIEW, and certain queries using UNION
Parallelized hash joins and parallelized sequential scans now perform better
Postgres is creating simple index more than 2 days. Where can I see a log, or something about status index creation? Before this situation I've created index couple times and it took about an hour or less. I have about 5.5m rows and execute next command:
CREATE INDEX collapse_url ON tablle (url, collapse)
There are two possibilities:
The CREATE INDEX statement is waiting for a lock.
You should be able to see that in the pg_stat_activity view.
If that is your problem, end all concurrent long running transactions (e.g. using pg_terminate_backend).
The CREATE INDEX statement is truly taking very long (unlikely with a few million rows).
In that case, you can speed up processing by increasing maintenance_work_mem before you create the index.
There have been relevant improvements in this area in recent versions:
PostgreSQL v11 introduced parallel index builds.
PostgreSQL v12 introduced the view pg_stat_progress_create_index to monitor the progress of CREATE INDEX.
PostgreSQL v12 also introduced CREATE INDEX CONCURRENTLY to avoid a long ACCESS EXCLUSIVE lock on the table.
I have a table which index size is too big (about 2G). When I restore the database to a VM, the size is only 200M so I need to rebuild/recreate the index and I will probably do this online.
What is the difference between re-building (reindex) and re-creating the index, and which one is better when I do it online? Particularly, which option allows querying the DB during the operation?
The REINDEX command requires an exclusive table lock, which means that it'll stall any accesses to the table until the command has completed. If you can afford that kind of maintenance window it's perfectly fine.
The alternative for online rebuilding is to create a new index using CREATE INDEX CONCURRENTLY, then drop the old one. This will take longer to complete, but allows access to the table while rebuilding the index.
Postgres 12 has added a REINDEX INDEX CONCURRENTLY command, which does what you want here. https://paquier.xyz/postgresql-2/postgres-12-reindex-concurrently/ https://www.depesz.com/2019/03/29/waiting-for-postgresql-12-reindex-concurrently/
We have the requirement to contain index size without locking tables. I tried to use 'create index concurrently ..' but it was resulting in INVALID index being created on one of the systems.We tried to do
- drop index
- drop index concurrently
- reindex table
However intermittently they were also getting stuck. This makes the whole approach of creating indexes concurrently via script is vulnerable.
Any ideas how this can be made full proof without manual intervention? If not, what are other effective ways to contain index sizes on postgreSQL in automated fashion on large and busy tables.
To benefit with concurrent build of indexes in script, you need to add logic in next commands (as you can't put in transaction). Simply check if index is not INVALID in next line and if it is, abort script.
Also I would reverse actions: first you build new index concurrently. If success, drop old one.
https://www.postgresql.org/docs/current/static/sql-createindex.html:
If a problem arises while scanning the table, such as a deadlock or a
uniqueness violation in a unique index, the CREATE INDEX command will
fail but leave behind an "invalid" index. This index will be ignored
for querying purposes because it might be incomplete; however it will
still consume update overhead.
and
Another difference is that a regular CREATE INDEX command can be
performed within a transaction block, but CREATE INDEX CONCURRENTLY
cannot.