I am trying to fetch blocks of rows from my Sybase DB (ie. fetch 100 rows at a time per transaction).
However, I have read here and here that T-SQL simply does not support this.
Is there any work around for this?
Also, if it is possible to fetch multiple rows at a time, how could I modify the following code to do so:
DECLARE my_cursor CURSOR FOR
SELECT
COL1,
COL2,
...
COLN
FROM
MY_DB
WHERE
SOME_CONDITION_SATISFIED
GO
DECLARE
VAR1 TYPE,
VAR2 TYPE,
...
VARN TYPE
SET NO COUNT ON
OPEN my_cursor
WHILE ##SQLSTATUS = 0
BEGIN
FETCH my_cursor into
#VAR1,
#VAR2,
...
#VARN
END
CLOSE my_cursor
DEALLOCATE CURSOR my_cursor
Any help would be appreciated.
To the database programmer, SQL Server treats cursors just like Sybase and there's nothing wrong with your code. In T-SQL you basically don't have control on the pace the server fetches physical rows to populate the cursor. To the programmer point of view it shouldn't matter though.
If you need this fine grained control over the network usage, you'd better use other APIs better suited for ETL process, such as SSIS.
If your problem is to prevent database contention because of an open transaction, though, you can use transaction isollation level read uncommitted (not to lock the table) or insert some transaction management logic with a counter, as:
....
DECLARE #counter INT; SET #counter = 0;
BEGIN TRANSACTION; -- OPEN THE FIRST TRANSACTION
OPEN my_cursor
WHILE ##SQLSTATUS = 0
BEGIN
-- control transactions: commit at every 100 and opens another transaction
IF #counter > 99
BEGIN
SET #counter = 0;
COMMIT TRANSACTION;
BEGIN TRANSACTION;
END
ELSE SET #counter = #counter +1;
FETCH my_cursor into
#VAR1,
#VAR2,
...
#VARN
END
COMMIT TRANSACTION; -- CLOSE THE LAST TRANSACTION
....
you can fetch multiple rows at time in sybase. use below to fetch 100 rows at a time
set cursor rows 100 for cursor_name
Related
I have million of rows in pg_largeobject_metadata table I want to delete. What I have tried so far is :
A simple select lo_unlink(oid) works fine
A perform lo_unlink(oid) in a loop of 10000 rows will also work fine
So when I delete recursively multiple rows i get this error. I cannot increase max_locks_per_transaction because it is managed by AWS.
ERROR: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
CONTEXT: SQL statement "SELECT lo_unlink(c_row.oid)" PL/pgSQL function inline_code_block line 21 at
PERFORM SQL state: 53200
Here is the program I tried to write but I still get the Out of shared memory ERROR.
DO $proc$
DECLARE
v_fetch bigInt;
v_offset bigInt;
nbRows bigInt;
c_row record;
c_rows CURSOR(p_offset bigInt, p_fetch bigInt) FOR SELECT oid FROM pg_largeobject_metadata WHERE oid BETWEEN 1910001 AND 2900000 OFFSET p_offset ROWS FETCH NEXT p_fetch ROWS ONLY;
BEGIN
v_offset := 0;
v_fetch := 100;
select count(*) into nbRows FROM pg_largeobject_metadata WHERE oid BETWEEN 1910001 AND 2900000;
RAISE NOTICE 'End loop nbrows = %', nbRows;
LOOP -- Loop the different cursors
RAISE NOTICE 'offseter = %', v_offset;
OPEN c_rows(v_offset, v_fetch);
LOOP -- Loop through the cursor results
FETCH c_rows INTO c_row;
EXIT WHEN NOT FOUND;
perform lo_unlink(c_row.oid);
END LOOP;
CLOSE c_rows;
EXIT WHEN v_offset > nbRows;
v_offset := v_offset + v_fetch; -- The next 10000 rows
END LOOP;
END;
$proc$;
I am using Pg 9.5
Can anyone has faced this issue and could help please?
Each lo_unlink() grabs a lock on the object it deletes. These locks are freed only at the end of the transaction, and they are capped by max_locks_per_transaction * (max_connections + max_prepared_transactions) (see Lock Management). By default max_locks_per_transaction is 64, and cranking it up by several order of magnitudes is not a good solution.
The typical solution is to move the outer LOOP from your DO block into your client-side code, and commit the transaction at each iteration (so each transaction removes 10000 large objects and commits).
Starting with PostgreSQL version 11, a COMMIT inside the DO block would be possible, just like transaction control in procedures is possible.
I have a database that gets populated daily with incremental data and then at the end of each month a full download of the month's data is put into the system. Our business wants each day put into the system and then at the end of the month the daily stuff is removed and the full month data is left. I have written the query below and if you could help I'd appreciate it.
DECLARE #looper INT
DECLARE #totalindex int;
select name, (substring(name,17,8)) as Attempt, substring(name,17,4) as [year], substring(name,21,2) as [month], create_date
into #work_to_do_for
from sys.databases d
where name like 'Snapshot%' and
d.database_id >4 and
(substring(name,21,2) = DATEPART(m, DATEADD(m, -1, getdate()))) AND (substring(name,17,4) = DATEPART(yyyy, DATEADD(m, -1, getdate())))
order by d.create_date asc
SELECT #totalindex = COUNT(*) from #work_to_do_for
SET #looper = 1 -- reset and reuse counter
WHILE (#looper < #totalindex)
BEGIN;
set #looper=#looper+1
END;
DROP TABLE #work_to_do_for;
I'd need to perform the purge on several tables.
Thanks in advance.
When I delete large numbers of records, I always do it in batches and off-hours so as not to use up resources during production processes. To accomplish this, you incorporate a loop and some testing to find the optimal number to delete at a time.
begin transaction del -- I always use transactions as a safeguard
declare #count int = 1
while #count > 0
begin
delete top (100000) t
from dbo.MyTable t -- JOIN if necessary
-- WHERE if necessary
set #count = ##ROWCOUNT
end
Run this manually (without the WHILE loop) 1 time with 100000 records in parenthesis and see what your execution time is. Write it down. Run it again with 200000 records. Check the time; write it down. Run it with 500000 records. What you're looking for is a trend in the execution time. As long as the time required to delete 100000 records is decreasing as you increase the batch size, keep increasing it. You might end at 500k, but this method will help you find the optimal number to delete per batch. Then, run it as a loop.
That being said, if you are literally deleting MILLIONS of records, it might make more sense to drop and recreate the table as long as you aren't going to interfere with other processes. If you needed to save some of the data, you could insert what you needed into a new table (eg MyTable_New), drop the original table (MyTable), and rename MyTable_New to MyTable.
The script you've posted iterating through with a while loop to delete the rows should be changed to a set-based operation if at all possible. Relational database engines excel at set-based operations like
Delete dbo.table WHERE yourcolumn = 5
as opposed to iterating through one at a time. Especially if it will be for "several million" rows as you indicated in the comments above.
#rwking where are you putting the COMMIT to the Transaction.. I mean are you keeping the all eligible Delete count in single Transaction and doing one final Commit?
I have the similar type of Requirement where I have to Delete in batches, and also track the number of count affected in the end.
My Sample Code is as Follows:
Declare #count int
Declare #deletecount int
set #count=0
While(1=1)
BEGIN
BEGIN TRY
BEGIN TRAN
DELETE TOP 1000 FROM --CONDITION
SET #COUNT = #COUNT+##ROWCOUNT
IF (##ROWCOUNT)=0
Break;
COMMIT
END CATCH
BEGIN CATCH
ROLLBACK;
END CATCH
END
set #deletecount=#COUNT
Above Code Works fine, but how to keep track of #deletecount if Rollback happens in one of the batch.
This is related to this question but slightly different, I have while loop that inserts records and I want it to continue even if some inserts fail. So, the insertrecords procedure inserts records, by doing a where on the temp table for top 50 rows at a time.
The problem is that it won't continue if any of the inserts inside the insertrecords fail? How can I modify the sql to continue with the next 50 rows, even if it fails for current 50 records. I guess is there something like try/catch exception handling in sybase?
SELECT id INTO #temp FROM myTable
-- Loop through the rows of the temp table
WHILE EXISTS(SELECT 1 FROM #temp)
BEGIN
BEGIN TRANSACTION
exec insertrecords
IF ##error = 0
begin
print 'commited'
commit
end
else
begin
print 'rolled back'
rollback
end
DELETE TOP 50 FROM #temp order by id
END
-- Drop the temp table.
DROP TABLE #temp
Try putting the content inside your while block inside try cactch.
NOTE: The below sample is in SQL, try similar code in sybase.
`WHILE(SOME CONDITION)
BEGIN --start of while block
BEGIN TRY-start of try block
--your code here
END TRY
BEGIN CATCH
PRINT ERR_MESSAGE();
END CATCH
END --end of while loop.
`
I have a database purge process that uses a stored procedure to delete records from a huge table based on Expire Date, it runs every 3 weeks and delete about 3 million records.
Currently it is taking about 5 hours to purge the data which is causing lot of problems. I know there are lot of efficient way to write the code, but I'm out of ideas, please help me to the right direction.
--Stored Procedure
CREATE PROCEDURE [dbo].[pa_Expire_StoredValue_By_Date]
#ExpireDate DateTime, #NumExpired int OUTPUT, #RunAgain int OUTPUT
AS
-- This procedure expires all the StoredValue records that have an ExpireDate less than or equal to the DeleteDate provided
-- and have QtyUsed<QtyEarned
-- invoked by DBPurgeAgent
declare #NumRows int
set nocount on
BEGIN TRY
BEGIN TRAN T1
set #RunAgain = 1;
select #NumRows = count(*) from StoredValue where ExpireACK = 1;
if #NumRows = 0
begin
set rowcount 1800; -- only delete 1800 records at a time
update StoredValue with (RowLock)
set ExpireACK = 1
where ExpireACK = 0
and ExpireDate < #ExpireDate
and QtyEarned > QtyUsed;
set #NumExpired=##RowCount;
set rowcount 0
end
else begin
set #NumExpired = #NumRows;
end
if #NumExpired = 0
begin -- stop processing when there are no rows left
set #RunAgain = 0;
end
else begin
Insert into SVHistory (LocalID, ServerSerial, SVProgramID, CustomerPK, QtyUsed, Value, ExternalID, StatusFlag, LastUpdate, LastLocationID, ExpireDate, TotalValueEarned, RedeemedValue, BreakageValue, PresentedCustomerID, PresentedCardTypeID, ResolvedCustomerID, HHID)
select
SV.LocalID, SV.ServerSerial, SV.SVProgramID, SV.CustomerPK,
(SV.QtyEarned-SV.QtyUsed) as QtyUsed, SV.Value, SV.ExternalID,
3 as StatusFlag, getdate() as LastUpdate,
-9 as LocationID, SV.ExpireDate, SV.TotalValueEarned,
0 as RedeemedValue,
((SV.QtyEarned-SV.QtyUsed)*SV.Value*isnull(SVUOM.UnitOfMeasureLimit, 1)),
PresentedCustomerID, PresentedCardTypeID,
ResolvedCustomerID, HHID
from
StoredValue as SV with (NoLock)
Left Join
SVUnitOfMeasureLimits as SVUOM on SV.SVProgramID = SVUOM.SVProgramID
where
SV.ExpireACK = 1
Delete from StoredValue with (RowLock) where ExpireACK = 1;
end
COMMIT TRAN T1;
END TRY
BEGIN CATCH
set #RunAgain = 0;
IF ##TRANCOUNT > 0 BEGIN
ROLLBACK TRAN T1;
END
DECLARE #ErrorMessage NVARCHAR(4000);
DECLARE #ErrorSeverity INT;
DECLARE #ErrorState INT;
SELECT
#ErrorMessage = ERROR_MESSAGE(),
#ErrorSeverity = ERROR_SEVERITY(),
#ErrorState = ERROR_STATE();
RAISERROR (#ErrorMessage, #ErrorSeverity, #ErrorState);
END CATCH
Why you're running with this logic makes no sense to me. It looks like you are batching by rerunning the stored proc over and over again. You really should just do it in a WHILE loop and use smaller batches within a single run of the stored proc. You also should run in smaller transactions, this will speed things up considerably. Arguably, the way this is written you don't need a transaction. You can resume since you are flagging every record.
It's also not clear why you are touching the table 3 times. You really shouldn't need to update a flag AND select the rows into a new table AND then delete them. You can just use an output clause to do this in one step if desired, but you need to clarify your logic to get help on that.
Also, why are you using ROWLOCK? Lock escalation is fine and makes things run faster (less memory holding locks). Are you running this while the system is live? If it's after hours, use TABLOCK instead.
This is some suggested pseudo-code you can flesh out. I recommend taking #BatchSize as a parameter. Also obviously missing is error handling, this is just the core for the delete logic.
WHILE 1=1
BEGIN
BEGIN TRAN
UPDATE TOP (#BatchSize) StoredValue
SET <whatever>
INSERT INTO SVHistory <insert statement>
DELETE FROM StoredValue WHERE ExpireAck=1
IF ##ROWCOUNT = 0
BEGIN
COMMIT TRAN
BREAK;
END
COMMIT TRAN
END
First look at what is causing it to be slow by looking at the execution oplan. Is it the insert statement or the delete?
Due to the calculations in the insert, I would suspect it is the slower part (unless you have triggers or cascade delete on the table). You could change the history table to have the columns you use in the calculation and allow nulls in the calculated fields. Now you can insert to that table more quickly and then do the delete. In a separate step in your job, you can then update the calculations. This would at least tie things up for a shorter period of time, but depending on how the history table is acccessed may or may not be possible.
Another out of the box possibility is to rename your table StoredValue to StoredValueRaw and create a view called StoredValue that only displays the active records. Then the job to delete the records could run every fifteen minutes or so and only delete a few records at a time. This might be far less disruptive to the users even if the actual deletes take longer. You still might need to put the records in the history table at the time they are identified as expired.
You should possibly rethink doing this process every three weeks as well, the fewer records you have to deal with the faster it will go.
Let's say I've got a trigger like this:
CREATE TRIGGER trigger1
ON [dbo].[table1]
AFTER UPDATE
AS
BEGIN
--declare some vars
DECLARE #Col1 SMALLINT
DECLARE #Col1 TINYINT
--declare cursor
DECLARE Cursor1 CURSOR FOR
SELECT Col1, Col2 FROM INSERTED
--do the job
OPEN Cursor1
FETCH NEXT FROM Cursor1 INTO #Col1, #Col2
WHILE ##FETCH_STATUS = 0
BEGIN
IF ...something...
BEGIN
EXEC myProc1 #param1 = #Col1, #Param2 = #Col2
END
ELSE
IF ...something else...
BEGIN
EXEC myProc2 #param1 = #Col1, #Param2 = #Col2
END
FETCH NEXT FROM Cursor1 INTO #Col1, #Col2
END
--clean it up
CLOSE Cursor1
DEALLOCATE Cursor1
END
I want to be sure that Cursor1 is always closed and deallocated. Even myProc1 or myProc2 fails.
Shall I use try/catch block?
You could use the CURSOR_STATUS() function.
if CURSOR_STATUS('global','cursor_name') >= 0
begin
close cursor_name
deallocate cursor_name
end
reference: http://msdn.microsoft.com/en-us/library/ms177609.aspx
Yes, use TRY/CATCH but make sure you deallocate etc after.
Unfortunately, there is no finally in SQL Server.
However, I suggest wrapping this in another try/catch
CREATE TRIGGER trigger1 ON [dbo].[table1] AFTER UPDATE
AS
BEGIN
--declare some vars
DECLARE #Col1 SMALLINT, #Col1 TINYINT
BEGIN TRY
--declare cursor
DECLARE Cursor1 CURSOR FOR
SELECT Col1, Col2 FROM INSERTED
--do the job
OPEN Cursor1
FETCH NEXT FROM Cursor1 INTO #Col1, #Col2
WHILE ##FETCH_STATUS = 0
BEGIN
IF ...something...
EXEC myProc1 #param1 = #Col1, #Param2 = #Col2
ELSE
IF ...something else...
EXEC myProc2 #param1 = #Col1, #Param2 = #Col2
FETCH NEXT FROM Cursor1 INTO #Col1, #Col2
END
END TRY
BEGIN CATCH
--do what you have to
END CATCH
BEGIN TRY
--clean it up
CLOSE Cursor1
DEALLOCATE Cursor1
END TRY
BEGIN CATCH
--do nothing
END CATCH
END
Whether a cursor in a trigger is a good idea is a different matter...
Ten years later, I figure I should add some information to this particular question.
There are two primary solutions to your problem. First, use a LOCAL cursor declaration:
DECLARE --Operation
Cursor1 -- Name
CURSOR -- Type
LOCAL READ_ONLY FORWARD_ONLY -- Modifiers
FOR -- Specify Iterations
SELECT Col1, Col2 FROM INSERTED;
This limits your particular cursor to only your active session, rather than the global context of the server, assuming no other action is calling into this cursor. Similar in principle is to use a Cursor Variable, which would look like this:
DECLARE #Cursor1 CURSOR;
SET #Cursor1 = CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR SELECT Col1, Col2 FROM INSERTED;
In using a cursor variable, you can always overwrite it at anytime using the SET syntax, in addition to managing the scope to being within your particular session like the previous example. By overwriting the cursor context, you effectively deallocate any past reference it had. That said, both of these approaches accomplish your original intention by linking the status of the cursor to the activity of the current connection.
This might leave a lingering lock if your app context is using Connection Pooling, in which case you should use the Try-Catch pattern as follows:
CREATE TRIGGER trigger1
ON [dbo].[table1]
AFTER UPDATE
AS
BEGIN
--declare some vars
DECLARE #Col1 SMALLINT;
DECLARE #Col2 TINYINT;
--declare cursor
DECLARE
Cursor1
CURSOR
LOCAL READ_ONLY FORWARD_ONLY
FOR
SELECT
Col1,
Col2
FROM
INSERTED;
--do the job
OPEN Cursor1;
BEGIN TRY
FETCH
NEXT
FROM
Cursor1
INTO
#Col1,
#Col2;
WHILE ##FETCH_STATUS = 0
BEGIN
IF -- my condition
EXEC myProc1 #param1 = #Col1, #Param2 = #Col2;
ELSE IF -- additional condition
EXEC myProc2 #param1 = #Col1, #Param2 = #Col2;
FETCH
NEXT
FROM
Cursor1
INTO
#Col1,
#Col2;
END;
END TRY
BEGIN CATCH
-- Error Handling
END CATCH
--clean it up
CLOSE Cursor1;
DEALLOCATE Cursor1;
END;
Using the pattern in this way reduces the code duplication, or need to check the status of the cursor. Basically, the Cursor-initialization should be safe, as is the open statement. Once the cursor is open, you will want to always close-deallocate it from the session, and that should always be a safe action assuming the cursor has been opened (which we just established should always be a safe operation). As such, leaving those outside the confines of the Try-Catch means that everything can be neatly closed at the end, after the Catch block.
It's worth mentioning that I specified the READ_ONLY attribute of the cursor, as well as FORWARD_ONLY, since your sample code didn't scroll back-and-forth between records in the set. If you are modifying the underlying rows in those procedures, you are probably better off using a STATIC cursor to ensure you don't accidentally cause an infinite loop. That shouldn't be a problem since you're using the INSERTED table to manage your cursor context, but still worth mentioning for other potential use cases.
If you want to learn more about cursors in SQL Server, I highly recommend reading this blog post on the subject, as he goes into great detail explaining what the various modifiers of a cursor are, and the effects they have within the Database Engine.
What you should do is never ever use a cursor in a trigger. Write correct set-based code instead. If someone did an import of data into your table of 100,000 new records you would lock up the table for hours and bring your database to a screaming halt. It is a very poor practice to use a cursor in a trigger.