I have a table DB.DATA_FEED that I update using a T/SQL Procedure. Every minute, the procedure below is executed 100 times for different data.
ALTER PROCEDURE [DB].[UPDATE_DATA_FEED]
#P_MARKET_DATE varchar(max),
#P_CURR1 int,
#P_CURR2 int,
#P_PERIOD float(53),
#P_MID float(53)
AS
BEGIN
BEGIN TRY
UPDATE DB.DATA_FEED
SET
MID = #P_MID,
MARKET_DATE = convert(datetime,#P_MARKET_DATE, 103)
WHERE
cast(MARKET_DATE as date) =
cast(convert(datetime,#P_MARKET_DATE, 103) as date) AND
CURR1 = #P_CURR1 AND
CURR2 = #P_CURR2 AND
PERIOD = #P_PERIOD
IF ##TRANCOUNT > 0
COMMIT WORK
END TRY
BEGIN CATCH
--error code
END CATCH
END
END
When Users use the application, then they also read from this table, as per the SQL below. Potentially this select can run thousands of times in one minute. (Questions marks are replaced by parser with appropriate date/numbers)
DECLARE #MYDATE AS DATE;
SET #MYDATE='?'
SELECT *
FROM DB.DATA_FEED
WHERE MARKET_DATE>=#MYDATE AND MARKET_DATE<DATEADD(D,1,#MYDATE)
AND CURR1 = ?
AND CURR2 = ?
AND PERIOD = ?
ORDER BY PERIOD
I have sometimes, albeit rarely, got a database lock.
Using the the script from http://sqlserverplanet.com/troubleshooting/blocking-processes-lead-blocker I saw it was SPID=58. I then did DECLARE #SPID INT; SET #SPID = 58; DBCC INPUTBUFFER(#SPID) to find the SQL script which turned out to be my select statement.
Is there something wrong with my SQL code? What can I do to prevent such locks happening in the future?
Thanks
Readers have priority over writers so when someone is writing the readers have to wait for the writing to finish. There are two Table Hints you ca try one is NOLOCK that reads uncommited lines (dirty reads) and the other is READPAST (only reads information that has been commited on the last commit). In both cases the readers never block the table, there for do not deadlock a writer.
Writers can block other writers but, if I understood correctly, only one write per execution so the readers will intercalate writes, diminuishing the deadlocks.
Hope it helps.
Related
We am doing a system redesigning and due to the change in design we need to import data from multiple similar source tables into one table. For this same, I am running a loop which have the list of tables and importing all the data. However, due to massive amount of data, I got out of memory error after execution of around 12 hours and 20 tables. Now I discovered that the loop runs in a single transaction which I don't need since the system which is filling the data is suspended for that time. Having this transaction thing, I believe, it is taking longer time also. My requirement is to run my query without any transaction.
DO $$DECLARE r record;
BEGIN
FOR r IN SELECT '
INSERT INTO dbo.tb_requests
(node_request_id, request_type, id, process_id, data, timestamp_d1, timestamp_d2, create_time, is_processed)
SELECT lpad(A._id, 32, ''0'')::UUID, (A.data_type + 1) request_type, B.id, B.order_id, data_value, timestamp_d1, timestamp_d2, create_time, TRUE
FROM dbo.data_store_' || id || ' A
JOIN dbo.tb_new_processes B
ON A.process_id = B.process_id
WHERE A._id != ''0'';
' as log_query FROM dbo.list_table
ORDER BY line_id
LOOP
EXECUTE r.log_query;
END LOOP;
END$$;
This is a sample code block. It is not the actual code block but I think, it will give the idea.
Error Message(Translation from Original Japanese error Message):
ERROR: Out of memory
DETAIL: Request for size 32 failed in memory context "ExprContext".
SQL state: 53200
You cannot to run any statement on server side without transaction. For some modern Postgres releases you can run commit statement inside DO statement. It is closes current transaction and starts new transactions. This can breaks very long transaction, and can solve the problem with memory leak - Postgres releasing some memory at transaction end.
Or use shell scripts instead (bash) if it is possible.
I've inherited a stored procedure that works perfectly when we are running single user, but when more than one user is running the procedure all hell breaks lose. I know this is poor structure but like I said, I inherited it and have to live with it.
Is there any way to flush the buffer after set line? I just want to make sure the number generated is always unique.
begin transaction;
try
update s
set s.NEXT_YMMPC = s.NEXT_YMMPC + 1
from system s;
#YMMPC_Key = (select top 1 cast(s.NEXT_YMMPC as sql_char(10)) from system s);
#YMMPC_Key = right('0000000'+trim(#YMMPC_Key),10);
insert into YMMProductCatalog (Year,MakeCode,ModelCode,ProductCode,CatalogCode,YMMPC_Key) values (#Year,#MakeCode,#ModelCode,#ProductCode,#CatalogCode,#YMMPC_Key);
commit work;
insert into __output select #YMMPC_Key from system.iota;
catch all
rollback work;
raise;
end;
TIA - Jay
I have a database that gets populated daily with incremental data and then at the end of each month a full download of the month's data is put into the system. Our business wants each day put into the system and then at the end of the month the daily stuff is removed and the full month data is left. I have written the query below and if you could help I'd appreciate it.
DECLARE #looper INT
DECLARE #totalindex int;
select name, (substring(name,17,8)) as Attempt, substring(name,17,4) as [year], substring(name,21,2) as [month], create_date
into #work_to_do_for
from sys.databases d
where name like 'Snapshot%' and
d.database_id >4 and
(substring(name,21,2) = DATEPART(m, DATEADD(m, -1, getdate()))) AND (substring(name,17,4) = DATEPART(yyyy, DATEADD(m, -1, getdate())))
order by d.create_date asc
SELECT #totalindex = COUNT(*) from #work_to_do_for
SET #looper = 1 -- reset and reuse counter
WHILE (#looper < #totalindex)
BEGIN;
set #looper=#looper+1
END;
DROP TABLE #work_to_do_for;
I'd need to perform the purge on several tables.
Thanks in advance.
When I delete large numbers of records, I always do it in batches and off-hours so as not to use up resources during production processes. To accomplish this, you incorporate a loop and some testing to find the optimal number to delete at a time.
begin transaction del -- I always use transactions as a safeguard
declare #count int = 1
while #count > 0
begin
delete top (100000) t
from dbo.MyTable t -- JOIN if necessary
-- WHERE if necessary
set #count = ##ROWCOUNT
end
Run this manually (without the WHILE loop) 1 time with 100000 records in parenthesis and see what your execution time is. Write it down. Run it again with 200000 records. Check the time; write it down. Run it with 500000 records. What you're looking for is a trend in the execution time. As long as the time required to delete 100000 records is decreasing as you increase the batch size, keep increasing it. You might end at 500k, but this method will help you find the optimal number to delete per batch. Then, run it as a loop.
That being said, if you are literally deleting MILLIONS of records, it might make more sense to drop and recreate the table as long as you aren't going to interfere with other processes. If you needed to save some of the data, you could insert what you needed into a new table (eg MyTable_New), drop the original table (MyTable), and rename MyTable_New to MyTable.
The script you've posted iterating through with a while loop to delete the rows should be changed to a set-based operation if at all possible. Relational database engines excel at set-based operations like
Delete dbo.table WHERE yourcolumn = 5
as opposed to iterating through one at a time. Especially if it will be for "several million" rows as you indicated in the comments above.
#rwking where are you putting the COMMIT to the Transaction.. I mean are you keeping the all eligible Delete count in single Transaction and doing one final Commit?
I have the similar type of Requirement where I have to Delete in batches, and also track the number of count affected in the end.
My Sample Code is as Follows:
Declare #count int
Declare #deletecount int
set #count=0
While(1=1)
BEGIN
BEGIN TRY
BEGIN TRAN
DELETE TOP 1000 FROM --CONDITION
SET #COUNT = #COUNT+##ROWCOUNT
IF (##ROWCOUNT)=0
Break;
COMMIT
END CATCH
BEGIN CATCH
ROLLBACK;
END CATCH
END
set #deletecount=#COUNT
Above Code Works fine, but how to keep track of #deletecount if Rollback happens in one of the batch.
I have big stored procedures that handle user actions.
They consist of multiple select statements. These are filtered, most of the times only getting one row. The Selects are copied into temptables or otherwise evaluated.
Finally, a merge-Statement does the needed changes in the DB.
All is encapsulated in a transaction.
I have concurrent input from users, and the selected rows of the select statements should be locked to keep data integrity.
How can I lock the selected Rows of all select statements, so that they aren't updated through other transactions while the current transaction is in process?
Does a table hint combination of ROWLOCK and HOLDLOCK work in a way that only the selected rows are locked, or are the whole tables locked because of the HOLDLOCK?
SELECT *
FROM dbo.Test
WITH (ROWLOCK HOLDLOCK )
WHERE id = #testId
Can I instead use
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
right after the start of the transaction? Or does this lock the whole tables?
I am using SQL2008 R2, but would also be interested if things work differently in SQL2012.
PS: I just read about the table hints UPDLOCK and SERIALIZE. UPDLOCK seems to be a solution to lock only one row, and it seems as if UPDLOCK always locks instead of ROWLOCK, which does only specify that locks are row based IF locks are applied. I am still confused about the best way to solve this...
Changing the isolation level fixed the problem (and locked on row level):
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
Here is how I tested it.
I created a statement in a blank page of the SQL Management Studio:
begin tran
select
*
into #message
from dbo.MessageBody
where MessageBody.headerId = 28
WAITFOR DELAY '0:00:05'
update dbo.MessageBody set [message] = 'message1'
where headerId = (select headerId from #message)
select * from dbo.MessageBody where headerId = (select headerId from #message)
drop table #message
commit tran
While executing this statement (which takes at last 5 seconds due to the delay), I called the second query in another window:
begin tran
select
*
into #message
from dbo.MessageBody
where MessageBody.headerId = 28
update dbo.MessageBody set [message] = 'message2'
where headerId = (select headerId from #message)
select * from dbo.MessageBody where headerId = (select headerId from #message)
drop table #message
commit tran
and I was rather surprised that it executed instantaneously. This was due to the default SQL Server transaction level "Read Commited" http://technet.microsoft.com/en-us/library/ms173763.aspx . Since the update of the first script is done after the delay, during the second script there are no umcommited changes yet, so the row 28 is read and updated.
Changing the Isolation level to Serialization prevented this, but it also prevented concurrency - both scipts were executed consecutively.
That was OK, since both scripts read and changed the same row (via headerId=28). Changing headerId to another value in the second script, the statements were executed parallel. So the lock from SERIALIZATION seems to be on row level.
Adding the table hint
WITH ( SERIALIZABLE)
in the first select of the first statement does also prevent further reads oth the selected row.
I have a database purge process that uses a stored procedure to delete records from a huge table based on Expire Date, it runs every 3 weeks and delete about 3 million records.
Currently it is taking about 5 hours to purge the data which is causing lot of problems. I know there are lot of efficient way to write the code, but I'm out of ideas, please help me to the right direction.
--Stored Procedure
CREATE PROCEDURE [dbo].[pa_Expire_StoredValue_By_Date]
#ExpireDate DateTime, #NumExpired int OUTPUT, #RunAgain int OUTPUT
AS
-- This procedure expires all the StoredValue records that have an ExpireDate less than or equal to the DeleteDate provided
-- and have QtyUsed<QtyEarned
-- invoked by DBPurgeAgent
declare #NumRows int
set nocount on
BEGIN TRY
BEGIN TRAN T1
set #RunAgain = 1;
select #NumRows = count(*) from StoredValue where ExpireACK = 1;
if #NumRows = 0
begin
set rowcount 1800; -- only delete 1800 records at a time
update StoredValue with (RowLock)
set ExpireACK = 1
where ExpireACK = 0
and ExpireDate < #ExpireDate
and QtyEarned > QtyUsed;
set #NumExpired=##RowCount;
set rowcount 0
end
else begin
set #NumExpired = #NumRows;
end
if #NumExpired = 0
begin -- stop processing when there are no rows left
set #RunAgain = 0;
end
else begin
Insert into SVHistory (LocalID, ServerSerial, SVProgramID, CustomerPK, QtyUsed, Value, ExternalID, StatusFlag, LastUpdate, LastLocationID, ExpireDate, TotalValueEarned, RedeemedValue, BreakageValue, PresentedCustomerID, PresentedCardTypeID, ResolvedCustomerID, HHID)
select
SV.LocalID, SV.ServerSerial, SV.SVProgramID, SV.CustomerPK,
(SV.QtyEarned-SV.QtyUsed) as QtyUsed, SV.Value, SV.ExternalID,
3 as StatusFlag, getdate() as LastUpdate,
-9 as LocationID, SV.ExpireDate, SV.TotalValueEarned,
0 as RedeemedValue,
((SV.QtyEarned-SV.QtyUsed)*SV.Value*isnull(SVUOM.UnitOfMeasureLimit, 1)),
PresentedCustomerID, PresentedCardTypeID,
ResolvedCustomerID, HHID
from
StoredValue as SV with (NoLock)
Left Join
SVUnitOfMeasureLimits as SVUOM on SV.SVProgramID = SVUOM.SVProgramID
where
SV.ExpireACK = 1
Delete from StoredValue with (RowLock) where ExpireACK = 1;
end
COMMIT TRAN T1;
END TRY
BEGIN CATCH
set #RunAgain = 0;
IF ##TRANCOUNT > 0 BEGIN
ROLLBACK TRAN T1;
END
DECLARE #ErrorMessage NVARCHAR(4000);
DECLARE #ErrorSeverity INT;
DECLARE #ErrorState INT;
SELECT
#ErrorMessage = ERROR_MESSAGE(),
#ErrorSeverity = ERROR_SEVERITY(),
#ErrorState = ERROR_STATE();
RAISERROR (#ErrorMessage, #ErrorSeverity, #ErrorState);
END CATCH
Why you're running with this logic makes no sense to me. It looks like you are batching by rerunning the stored proc over and over again. You really should just do it in a WHILE loop and use smaller batches within a single run of the stored proc. You also should run in smaller transactions, this will speed things up considerably. Arguably, the way this is written you don't need a transaction. You can resume since you are flagging every record.
It's also not clear why you are touching the table 3 times. You really shouldn't need to update a flag AND select the rows into a new table AND then delete them. You can just use an output clause to do this in one step if desired, but you need to clarify your logic to get help on that.
Also, why are you using ROWLOCK? Lock escalation is fine and makes things run faster (less memory holding locks). Are you running this while the system is live? If it's after hours, use TABLOCK instead.
This is some suggested pseudo-code you can flesh out. I recommend taking #BatchSize as a parameter. Also obviously missing is error handling, this is just the core for the delete logic.
WHILE 1=1
BEGIN
BEGIN TRAN
UPDATE TOP (#BatchSize) StoredValue
SET <whatever>
INSERT INTO SVHistory <insert statement>
DELETE FROM StoredValue WHERE ExpireAck=1
IF ##ROWCOUNT = 0
BEGIN
COMMIT TRAN
BREAK;
END
COMMIT TRAN
END
First look at what is causing it to be slow by looking at the execution oplan. Is it the insert statement or the delete?
Due to the calculations in the insert, I would suspect it is the slower part (unless you have triggers or cascade delete on the table). You could change the history table to have the columns you use in the calculation and allow nulls in the calculated fields. Now you can insert to that table more quickly and then do the delete. In a separate step in your job, you can then update the calculations. This would at least tie things up for a shorter period of time, but depending on how the history table is acccessed may or may not be possible.
Another out of the box possibility is to rename your table StoredValue to StoredValueRaw and create a view called StoredValue that only displays the active records. Then the job to delete the records could run every fifteen minutes or so and only delete a few records at a time. This might be far less disruptive to the users even if the actual deletes take longer. You still might need to put the records in the history table at the time they are identified as expired.
You should possibly rethink doing this process every three weeks as well, the fewer records you have to deal with the faster it will go.