I am using Postgresql 9.6.
I have a stored proc which is called by Scala. This stored proc is a wrapper, i.e. it will call another stored procs for each input list passed in wrapper. For e.g. wrapper has input list of 100 elements, so the internal stored proc will be called 100 times per element.
The internal proc is data heavy proc, which creates 4-5 temp tables and processes the data and returns.
So wrapper will collect all the data and finally complete.
get_data_synced(date, text, integer[])
Here the text is comma-separated items (10-1000 depending on use -case).
Basically the problem is if I pass a bigger number 100-200 items i.e. in a loop we call the internal procs that many times, it throws the error:
SQL execution failed (Reason: ERROR: out of shared memory
Hint: You might need to increase max_locks_per_transaction.
I understand that create temp table inside the internal function will create locks. Bu each time the proc is called, first thing is DROP and then CREATE the temp table.
DROP TABLE IF EXISTS _temp_data_1;
CREATE TEMP TABLE _temp_data_1 AS (...);
DROP TABLE IF EXISTS _temp_data_2;
CREATE TEMP TABLE _temp_data_2 AS (...);
..
..
..
So even if the proc is called 1000 times, the first thing it does is drop table (which should release locks?) and then create the table.
The max_locks_per_transaction is set to 256.
Now, the transaction is not over until my wrapper function (outside function) is over, right?
So it means that even if I am dropping the temp table, the locks are not released?
Is there a way to release the lock on temp table immediately once my function is complete?
You diagnosis is correct, the locks survive until the end of the transaction. Even if it was dropped in the same transaction that created it, and even if it is a temp table. Perhaps this could be optimized, but that is currently how it works.
For work-arounds, why not just truncate the table, rather than drop and re-create it, if it already exists?
Related
We have this pair of trigger and function that we use on our psql database for the longest time. Basically, the trigger is called each time there is a new record to the main table, and each row is inserted to the monthly partition individually. Following is the trigger function:
CREATE TRIGGER partition_mic_teams_endpoint_trg1
BEFORE INSERT ON "mic_teams_endpoint"
FOR EACH ROW EXECUTE
PROCEDURE trg_partition_mic_teams_endpoint('month');
The function we have creates monthly partitions based on a timestamp field in each row.
I have two questions:
List item Even if I try to COPY a bunch of rows from CSV to the main table, is this trigger/function going to insert each row individually? Is this efficient?
If that is the case, is it possible to have support for COPYing data to partitions instead of INSERT.
Thanks,
Note: I am sorry if I did not provide enough information for an answer
Yes, a row level trigger will be called for each row separately, and that will make COPY quite a bit slower.
One thing you could try is a statement level AFTER trigger that uses a transition table, so that you can
INSERT INTO destination SELECT ... FROM transition_table;
That should be faster, but you should test it to be certain.
See the documentation for details.
I'm trying to obtain an indefinite lock on my Postgresql database (specifically on a table called orders) for QA purposes. In short, I want to know if certain locks on a table prevent or indefinitely block database migrations for adding columns (I think ALTER TABLE grabs an ACCESS EXCLUSIVE LOCK).
My plan is to:
grab a table lock or a row lock on the orders table
run the migration to add a column (an ALTER TABLE statement that grabs an ACCESS EXCLUSIVE LOCK)
issue a read statement to see if (2) is blocked (the ACCESS EXCLUSIVE LOCK blocks reads, and so this would be a problem that I'm trying to QA).
How would one do this? How do I grab a row lock on a table called orders via the Rails Console? How else could I do this?
Does my plan make sense?
UPDATE
It turns out open row-level transactions actually do block ALTER TABLE statements that grab an ACCESS EXCLUSIVE LOCK like table migrations that add columns. For example, when I run this code in one process:
Order.first.with_lock do
binding.pry
end
It blocks my migration in another process to add a column to the orders table. That migration's ACCESS EXCLUSIVE LOCK blocks all reads and select statements to the orders table, causing problems for end users.
Why is this?
Let's say you're in a transaction, selecting rows from a table with various where clauses. Halfway through, some other transaction adds a column to that table. Now you are getting back more fields than you did previously. How is your application supposed to handle this?
I am using postgresql 8.4 in backend. In backend I made a postgres function to get some data. The function won't write any data to DB so just read data from other tables. The function internally will create a temp table then return a set of records.
When I monitoring the server I found out this function is blocking other connections doing data insertion.
So just wondering creating temp table could block data insertion from other connections?
Further question. I have a function A, inside this function there is a function call to B, function B will return a set of records (the record has two columns say "name","email"). Then I need a variable in function A to hold the data returned from function B. When data returned from B I will use the first column to do something then use the second col to other staff.
So at the moment I use temp table to hold the returned setof records from function B. Is there another way to hold returned records?
Found this not sure helpful or not:
https://wiki.postgresql.org/wiki/Return_more_than_one_row_of_data_from_PL/pgSQL_functions
You should def avoid creating such kind of functions, as in multi-session environment, or even just using multi-row requests from the same user session, server will try to create the temp table per each user session + per each queried row.
As multiple requests need to create the same resource (your temp table) - sure, server can handle just one row of one request at the same time.
See more about postgresql explicit-locking
Consider using view instead of creating temp table.
P.S. Your statement that query does not write any data, just reads, is not true. Query actually does DDL and DML operations during its execution.
In postgresql: multiple sessions want to get one record from the the table, but we need to make sure they don't interfere with each other. I could do it using message queue: put the data in a queue, and them let each session get data from the queue. But is it doable in postgresql? since it will be easier for SQL guys to cal stored procedure. Any way to configure a stored procedure so that no concurrent calling will happen, or use some special lock?
I would recommend making sure the stored procedure uses SELECT FOR UPDATE, which should prevent the same row in the table from being accessed by multiple transactions.
Per the Postgres doc:
FOR UPDATE causes the rows retrieved by the SELECT statement to be
locked as though for update. This prevents them from being modified or
deleted by other transactions until the current transaction ends. That
is, other transactions that attempt UPDATE, DELETE, SELECT FOR UPDATE,
SELECT FOR NO KEY UPDATE, SELECT FOR SHARE or SELECT FOR KEY SHARE of
these rows will be blocked until the current transaction ends. The FOR
UPDATE lock mode is also acquired by any DELETE on a row, and also by
an UPDATE that modifies the values on certain columns. Currently, the
set of columns considered for the UPDATE case are those that have an
unique index on them that can be used in a foreign key (so partial
indexes and expressional indexes are not considered), but this may
change in the future.
More SELECT info.
So you don't end up locking all of the rows in the table at once (i.e. by SELECTing all of the records), I would recommend you use ORDER BY to sort the table in a consistent manner, and then do a LIMIT 1, so that it only gets the next one in the queue. Also add a WHERE clause that checks for a certain column value (i.e. processed), and then once processed set the column to a value that will prevent the WHERE clause from picking it up.
I have a couple of questions regarding SqlBulkCopy.
I need to insert a few million records to a table from my business component. I am considering the following approaches:
Use SqlBulkCopy to insert data directly into the destination table. Because the table has existing data and index (which I cannot change), so I won't get bulk logging behavior and cannot apply TabLock.
Use SqlBulkCopy to insert data into a heap in temp db in one go (batchsize = 0). Once completed, use a stored procedure to move data from temp table to destination table.
Use SqlBulkCopy to insert data into a heap in temp db but specify a batchsize. Once completed, use a stored procedure to move data from temp table to destination table.
Split data and use multiple SqlBulkCopy to insert into multiple heaps in temp db concurrently. After each chunk of data is uploaded, use a stored procedure to move data from temp table to destination table.
Which approach has shortest end to end time?
If I use SqlBulkCopy to upload data into a table with index, will I be able to query the table at the same time?