My ERP Vendor has the following trigger on a table:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TRIGGER [dbo].[SOItem_DeleteCheck]
ON [dbo].[soitem]
FOR DELETE
AS
BEGIN
DECLARE #RecCnt int, #LogInfo varchar(256)
SET #RecCnt = (SELECT COUNT(*) FROM deleted)
IF #RecCnt > 150
BEGIN
RAISERROR (54010, 18, 1, 'SOItem') WITH LOG
ROLLBACK TRANSACTION
END
SET #LogInfo = 'Deleting ' + LTRIM(STR(#RecCnt)) + ' Rows From SOItem'
EXEC LogDeletes #LogInfo
END
GO
This seems very inefficient to me. Doesn't select count(*) take longer than Count(specific field)?
Honestly even if is is slower, I can run a select stament like that in less than a millisecond on my largest table that has millions of rows which this trigger is unlikely to hit. There is no real performance gain from changing it. I'm curious as to why you would want to rollback any transaction with more than 150 records thoug.
I think in the past there was a benefit to count(1) vs count(*), and we were all taught to use that approach, but at this point it's more about style than performance.
Related
I'm trying to understand how to lock a row, and only release that lock later.
I have a table like this :
create table testTable (Name varchar(100));
Some test data
insert into testTable (name) select 'Bob';
insert into testTable (name) select 'John';
insert into testTable (name) select 'Steve';
Now, I want to select one of those rows, and prevent other other queries from seeing this row. I achieve that like this :
begin transaction;
select * from testTable where name = 'Bob' for update;
In another window, I do this :
select * from testTable for update skip locked;
Great, I don't see 'Bob' in that result set. Now, I want to do something with the primary retrieved row (Bob), and after I did my work, I want to release that row again. Simple answer would be to do :
commit transaction
However, I am running multiple transactions on the same connection, so I can't just begin and commit transactions all over the show. Ideally I would like to have a "named" transaction, something like :
begin transaction 'myTransaction';
select * from testTable where name = 'Bob' for update;
//do stuff with the data, outside sql then later call ...
commit transaction 'myTransaction';
But postgres doesn't support that. I have found "prepare transaction", but that seems to be a pear-shaped path I don't want to go down, especially as these transaction seem to persist through restarts even.
Is there anyway I can have a reference to commit/rollback for a specific transaction?
You can have only one transaction in a database session, so the question as such is moot.
But I assume that you do not really want to run a transaction, you want to block access to a certain row for a while.
It is usually not a good idea to use regular database locks for such a purpose (the exception are advisory locks, which serve exactly that purpose, but are not tied to table rows). The problem is that long database transactions keep autovacuum from doing its job.
I recommend that you add a status column to the table and change the status rather than locking the row. That would server the same purpose in a more natural fashion and make your problem go away.
If you are concerned that the status flag might not get cleared due to application logic problems, replace it with a visible_from column of type timestamp with time zone that initially contains -infinity. Instead of locking the row, set the value to current_timestamp + INTERVAL '5 minutes'. Only select rows that fulfill WHERE visible_from < current_timestamp. That way the “lock” will automatically expire after 5 minutes.
In a ms sql database I have a table named combo where multiple inserts, updates and deletes can happen (as well as single, of course). In another table named migrimi_temp I keep track of these changes in the form of queries (query that would have to be executed in mysql to achieve the same result).
For example, if a delete query is performed for all rows where id > 50, the trigger should activate to store the following query into the log table:
DELETE FROM combo where id > 50;
Therefore this one delete query in the combo table would result in one row in the log table.
But if instead I have an insert query inserting 2 rows, a trigger should activate to store each insert into the log table. So this one insert query in the combo table would result into 2 new rows in the log table.
I intend to handle insert, update and delete actions into separated triggers. I had managed to write triggers for single row insert / update/ delete. Then it occurred to me that multiple actions might be performed too.
This is my attempt to handle the case of multiple inserts in one single query. I resorted to using cursors after not being able to adapt the initial trigger without a cursor. The trigger is executed successfully, but when I perform an insert (single or multiple rows) the execution hangs up indefinitely, or at least longer than reasonable .
USE [migrimi_test]
GO
/****** Object: Trigger [dbo].[c_combo] Script Date: 12/11/2017 5:33:46 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
create TRIGGER [dbo].[u_combo]
ON [migrimi_test].[dbo].[combo]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
DECLARE #c_id INT;
DECLARE #c_name nvarchar(100);
DECLARE #c_duration int;
DECLARE #c_isavailable INT;
DECLARE c CURSOR FOR
SELECT id, name, duration, isvisible FROM inserted
OPEN c
FETCH NEXT FROM c INTO #c_id, #c_name, #c_duration, #c_isavailable
WHILE ##FETCH_STATUS = 0
INSERT INTO [migrimi_temp].[dbo].[sql_query] (query)
VALUES ('INSERT INTO combo (id, name, duration, value, isavailable, createdAt, updatedAt) values ('+CAST(#c_id as nvarchar(50))+', '+'"'+#c_name+'"'+',
'+CAST(#c_duration as nvarchar(50))+', 1, '+CAST(#c_isavailable as nvarchar(50))+', Now(), Now());' )
FETCH NEXT FROM c INTO #c_id, #c_name, #c_duration, #c_isavailable
CLOSE c
END
DEALLOCATE c
GO
SQL server version is 2012. OS is windows server 2008 (though I doubt that is relevant). I was based mainly on these two resources: https://social.msdn.microsoft.com/Forums/sqlserver/en-US/40f5635c-9034-4e9b-8fd5-c02cec44ce86/how-to-let-trigger-act-for-each-row?forum=sqlgetstarted
and How can I get a trigger to fire on each inserted row during an INSERT INTO Table (etc) SELECT * FROM Table2?
This is part of a larger idea I am trying to accomplish, and until 2 days ago I was totally unfamiliar with triggers. I am trying to balance learning with accomplishing in reasonable amounts of time, but not doing so great
Cursors are notoriously slow in SQL Server.
Instead of using a cursor to loop over the inserted table, you can use insert...select which is a set based approach. It is much faster and is the recommended way to work in SQL:
CREATE TRIGGER [dbo].[u_combo]
ON [migrimi_test].[dbo].[combo]
AFTER INSERT
AS
BEGIN
INSERT INTO [migrimi_temp].[dbo].[sql_query] (query)
SELECT 'INSERT INTO combo (id, name, duration, value, isavailable, createdAt, updatedAt) values ('+CAST(id as nvarchar(50))+', "'+ name +'",
'+ CAST(duration as nvarchar(50)) +', 1, '+ CAST(isvisible as nvarchar(50))+ ', Now(), Now());'
FROM inserted
END
GO
I have a database that gets populated daily with incremental data and then at the end of each month a full download of the month's data is put into the system. Our business wants each day put into the system and then at the end of the month the daily stuff is removed and the full month data is left. I have written the query below and if you could help I'd appreciate it.
DECLARE #looper INT
DECLARE #totalindex int;
select name, (substring(name,17,8)) as Attempt, substring(name,17,4) as [year], substring(name,21,2) as [month], create_date
into #work_to_do_for
from sys.databases d
where name like 'Snapshot%' and
d.database_id >4 and
(substring(name,21,2) = DATEPART(m, DATEADD(m, -1, getdate()))) AND (substring(name,17,4) = DATEPART(yyyy, DATEADD(m, -1, getdate())))
order by d.create_date asc
SELECT #totalindex = COUNT(*) from #work_to_do_for
SET #looper = 1 -- reset and reuse counter
WHILE (#looper < #totalindex)
BEGIN;
set #looper=#looper+1
END;
DROP TABLE #work_to_do_for;
I'd need to perform the purge on several tables.
Thanks in advance.
When I delete large numbers of records, I always do it in batches and off-hours so as not to use up resources during production processes. To accomplish this, you incorporate a loop and some testing to find the optimal number to delete at a time.
begin transaction del -- I always use transactions as a safeguard
declare #count int = 1
while #count > 0
begin
delete top (100000) t
from dbo.MyTable t -- JOIN if necessary
-- WHERE if necessary
set #count = ##ROWCOUNT
end
Run this manually (without the WHILE loop) 1 time with 100000 records in parenthesis and see what your execution time is. Write it down. Run it again with 200000 records. Check the time; write it down. Run it with 500000 records. What you're looking for is a trend in the execution time. As long as the time required to delete 100000 records is decreasing as you increase the batch size, keep increasing it. You might end at 500k, but this method will help you find the optimal number to delete per batch. Then, run it as a loop.
That being said, if you are literally deleting MILLIONS of records, it might make more sense to drop and recreate the table as long as you aren't going to interfere with other processes. If you needed to save some of the data, you could insert what you needed into a new table (eg MyTable_New), drop the original table (MyTable), and rename MyTable_New to MyTable.
The script you've posted iterating through with a while loop to delete the rows should be changed to a set-based operation if at all possible. Relational database engines excel at set-based operations like
Delete dbo.table WHERE yourcolumn = 5
as opposed to iterating through one at a time. Especially if it will be for "several million" rows as you indicated in the comments above.
#rwking where are you putting the COMMIT to the Transaction.. I mean are you keeping the all eligible Delete count in single Transaction and doing one final Commit?
I have the similar type of Requirement where I have to Delete in batches, and also track the number of count affected in the end.
My Sample Code is as Follows:
Declare #count int
Declare #deletecount int
set #count=0
While(1=1)
BEGIN
BEGIN TRY
BEGIN TRAN
DELETE TOP 1000 FROM --CONDITION
SET #COUNT = #COUNT+##ROWCOUNT
IF (##ROWCOUNT)=0
Break;
COMMIT
END CATCH
BEGIN CATCH
ROLLBACK;
END CATCH
END
set #deletecount=#COUNT
Above Code Works fine, but how to keep track of #deletecount if Rollback happens in one of the batch.
I have a database purge process that uses a stored procedure to delete records from a huge table based on Expire Date, it runs every 3 weeks and delete about 3 million records.
Currently it is taking about 5 hours to purge the data which is causing lot of problems. I know there are lot of efficient way to write the code, but I'm out of ideas, please help me to the right direction.
--Stored Procedure
CREATE PROCEDURE [dbo].[pa_Expire_StoredValue_By_Date]
#ExpireDate DateTime, #NumExpired int OUTPUT, #RunAgain int OUTPUT
AS
-- This procedure expires all the StoredValue records that have an ExpireDate less than or equal to the DeleteDate provided
-- and have QtyUsed<QtyEarned
-- invoked by DBPurgeAgent
declare #NumRows int
set nocount on
BEGIN TRY
BEGIN TRAN T1
set #RunAgain = 1;
select #NumRows = count(*) from StoredValue where ExpireACK = 1;
if #NumRows = 0
begin
set rowcount 1800; -- only delete 1800 records at a time
update StoredValue with (RowLock)
set ExpireACK = 1
where ExpireACK = 0
and ExpireDate < #ExpireDate
and QtyEarned > QtyUsed;
set #NumExpired=##RowCount;
set rowcount 0
end
else begin
set #NumExpired = #NumRows;
end
if #NumExpired = 0
begin -- stop processing when there are no rows left
set #RunAgain = 0;
end
else begin
Insert into SVHistory (LocalID, ServerSerial, SVProgramID, CustomerPK, QtyUsed, Value, ExternalID, StatusFlag, LastUpdate, LastLocationID, ExpireDate, TotalValueEarned, RedeemedValue, BreakageValue, PresentedCustomerID, PresentedCardTypeID, ResolvedCustomerID, HHID)
select
SV.LocalID, SV.ServerSerial, SV.SVProgramID, SV.CustomerPK,
(SV.QtyEarned-SV.QtyUsed) as QtyUsed, SV.Value, SV.ExternalID,
3 as StatusFlag, getdate() as LastUpdate,
-9 as LocationID, SV.ExpireDate, SV.TotalValueEarned,
0 as RedeemedValue,
((SV.QtyEarned-SV.QtyUsed)*SV.Value*isnull(SVUOM.UnitOfMeasureLimit, 1)),
PresentedCustomerID, PresentedCardTypeID,
ResolvedCustomerID, HHID
from
StoredValue as SV with (NoLock)
Left Join
SVUnitOfMeasureLimits as SVUOM on SV.SVProgramID = SVUOM.SVProgramID
where
SV.ExpireACK = 1
Delete from StoredValue with (RowLock) where ExpireACK = 1;
end
COMMIT TRAN T1;
END TRY
BEGIN CATCH
set #RunAgain = 0;
IF ##TRANCOUNT > 0 BEGIN
ROLLBACK TRAN T1;
END
DECLARE #ErrorMessage NVARCHAR(4000);
DECLARE #ErrorSeverity INT;
DECLARE #ErrorState INT;
SELECT
#ErrorMessage = ERROR_MESSAGE(),
#ErrorSeverity = ERROR_SEVERITY(),
#ErrorState = ERROR_STATE();
RAISERROR (#ErrorMessage, #ErrorSeverity, #ErrorState);
END CATCH
Why you're running with this logic makes no sense to me. It looks like you are batching by rerunning the stored proc over and over again. You really should just do it in a WHILE loop and use smaller batches within a single run of the stored proc. You also should run in smaller transactions, this will speed things up considerably. Arguably, the way this is written you don't need a transaction. You can resume since you are flagging every record.
It's also not clear why you are touching the table 3 times. You really shouldn't need to update a flag AND select the rows into a new table AND then delete them. You can just use an output clause to do this in one step if desired, but you need to clarify your logic to get help on that.
Also, why are you using ROWLOCK? Lock escalation is fine and makes things run faster (less memory holding locks). Are you running this while the system is live? If it's after hours, use TABLOCK instead.
This is some suggested pseudo-code you can flesh out. I recommend taking #BatchSize as a parameter. Also obviously missing is error handling, this is just the core for the delete logic.
WHILE 1=1
BEGIN
BEGIN TRAN
UPDATE TOP (#BatchSize) StoredValue
SET <whatever>
INSERT INTO SVHistory <insert statement>
DELETE FROM StoredValue WHERE ExpireAck=1
IF ##ROWCOUNT = 0
BEGIN
COMMIT TRAN
BREAK;
END
COMMIT TRAN
END
First look at what is causing it to be slow by looking at the execution oplan. Is it the insert statement or the delete?
Due to the calculations in the insert, I would suspect it is the slower part (unless you have triggers or cascade delete on the table). You could change the history table to have the columns you use in the calculation and allow nulls in the calculated fields. Now you can insert to that table more quickly and then do the delete. In a separate step in your job, you can then update the calculations. This would at least tie things up for a shorter period of time, but depending on how the history table is acccessed may or may not be possible.
Another out of the box possibility is to rename your table StoredValue to StoredValueRaw and create a view called StoredValue that only displays the active records. Then the job to delete the records could run every fifteen minutes or so and only delete a few records at a time. This might be far less disruptive to the users even if the actual deletes take longer. You still might need to put the records in the history table at the time they are identified as expired.
You should possibly rethink doing this process every three weeks as well, the fewer records you have to deal with the faster it will go.
What would the processing load concern be if I had an "After Insert" trigger created on a table and in that trigger I performed a While loop to iterate through "potentially" multiple rows?
End result is I will 99.999% of the time have only 1 row, but as the future is unpredictable i also want to be able to handle multiple rows being inserted.
Trigger Model:
1) Insert information into the table
2) Create views specific to the client, via stored procedures (if possible)
What Say You? :)
Haven't fully developed but this is the design i am looking for, may not be structurally sound but should get the point acrossed.
CREATE TRIGGER dbo.New_Client_Setup
ON dbo.client
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON;
--Fill Temp Table
select * into #clients
from inserted
--Iterate through Temp Table
While (select count(*) from #clients) <> 0 BEGIN
declare #id int, #clnt nvarchar(10)
select top(1)
#id = id
, #clnt = short
order by id desc
Execute dbo.sp_Create_View_Client ( #id, #clnt )
-- Drop used ID
delete from #clients
where id = #id
END
Drop table #clients
END
GO
Again, observe the design of the trigger not necessarily the syntactic sugar
Design wise, reading the comments, I think you do not neccesarily need to do this in triggers. I would say you should do it as part of your insert statement in transactions - i.e. do the insert, and then do the loop that you want to do (whatever that does - execute dbo.sp_Create_View_Client)...
The second thing I would mention is what exactly is dbo.sp_Create_View_Client doing - is it a must-dependent on the insert? Meaning, what happens if the insert works fine, and the trigger fails? I would maybe do the whole insert and execute of the SP all in one transaction, so as to preserve data integrity.