I need to submit a series of queries to an Oracle server over ODBC from an MS SQL server and store the results as a table on the MS SQL server.
It has to be a pass through because the query requires a server side function defined on the Oracle server.
I can't save the table on the Oracle server and then access it via ODBC because of licensing restrictions from the vendor of the db running on Oracle.
Here's the code that returns the correct results, but I don't know how to save them:
DECLARE #BibID AS bigint
DECLARE BibList CURSOR FOR
SELECT BIB_ID FROM tblActiveSerialsThatHave740s
OPEN BibList
FETCH NEXT FROM BibList INTO #BibID
WHILE ##FETCH_STATUS=0
BEGIN
EXECUTE
('SELECT
AMDB.BIB_DATA.BIB_ID As BIB_ID,
AMDB.GetAllBibTag(AMDB.BIB_DATA.BIB_ID, ''740'', ''2'') As F740_All
FROM
AMDB.BIB_DATA
WHERE
AMDB.BIB_DATA.BIB_ID = ' + #BibID + '
GROUP BY BIB_ID '
)
AT REPORT
FETCH NEXT FROM BibList INTO #BibID
END
DEALLOCATE BibList
You need to use INSERT INTO to capture the results of an EXECUTE.
Because you are executing a passthrough query, the Distributed Transaction Coordinator is going to come into play and you might need to ensure that a distributed transaction is not created (it's unlikely to be necessary to have a distributed transaction in your case) or that the Distributed Transaction Coordinator service is running:
http://blogs.msdn.com/b/sqlprogrammability/archive/2008/08/22/how-to-create-an-autonomous-transaction-in-sql-server-2008.aspx
http://technet.microsoft.com/en-us/library/ms178532.aspx
http://www.sqlservercentral.com/Forums/Topic861249-392-1.aspx
Related
I'm trying to execute and insert query to a linked server in SQL Server.
For that I'm using INSERT INTO OPENQUERY statement.
The linked server is an Apache HIVE using Cloudera ODBC Provider.
The insert operation takes around 1 minute in my setup when performed from HIVE client.
However, SQL INSERT always times out after 30 seconds.
I set the Query Timeout parameter to 0 but it seems to be not affecting INSERT statement, however, it is working fine for SELECT statements taking longer time.
Is this a known limitation?
Is there a way to change the timeout for the insert statement when using OPENQUERY?
EDIT
I would like to clarify the setup I'm working with.
---------- ---------------------- ---------------
| MS SQL | => Linked Server => | Hive ODBC Provider | => | Hive Server |
---------- ---------------------- ---------------
In Hive, I have a table called calc_result where I would like to periodically store calculation results from the SQL server. For example, I try to insert using a query like this.
insert openquery(HIVE, 'select timestamp timestamp , tag tag, value value from calc_result')
values('2019-04-22 11:50:41', 'test',2.0)
The insert operation is captured correctly by HIVE server and a MapReduce job starts. However, the job will be killed after 30 seconds due to timeout.
The SQL server will show the below error message.
OLE DB provider "MSDASQL" for linked server "HIVE" returned message "[Cloudera][Hardy] (72) Query execution timeout expired.".
However, SELECT OPENQUERY works fine and would follow Query Timeout settings of the linked server (Which is set to 0 in this case).
Edit that is completely different use case from what I've imagined. In that case there should not be any difference in select/insert.
As you have configured your linked server timeout, there is a second place in the linked server properties you can check a Command Timeout setting in the provider string:
Other option that comes into my mind is instance wide timout. Default set for 600 seconds (10 minutes) which is way above your 30 seconds. However, you can still try it to see if there is any impact.
For infinite wait:
sp_configure 'show advanced options',1
go
reconfigure
go
sp_configure 'remote query timeout (s)',0
go
reconfigure
go
I would try using SELECT INTO temporary table and then materializing it using regular INSERT INTO:
SELECT c1, c2
INTO #temp_tab
FROM OPENQUERY(mylinkedserver, 'SELECT c1, c2 FROM remote_table');
INSERT INTO normal_table(col1, col2)
SELECT c1, c2
FROM #temp_tab;
EDIT:
You could try wrapping it with transaction and remove aliases:
BEGIN TRAN;
insert openquery(HIVE, 'select timestamp, tag, value from calc_result')
values('2019-04-22 11:50:41', 'test',2.0);
COMMIT;
If necessary set up DTC: How can I enable distributed transactions for a linked server?
While I didn't find a way to change OPENQUERYtimeout from 30 seconds, I found that using EXEC AT Linked Server to work fine for INSERT queries while adhering to timeout settings.
I accidentally stumbled upon the solution in this 2009 blog post. Databases might not be my strength, but I feel SQL Server documentation can be improved. A simple page that lists possible ways to interact with a Linked Server could've saved me lots of retries.
I want to use dblink in PL/pgSQL stored procedure in such way:
PERFORM dblink_exec('myconn', 'BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE');
PERFORM dblink_exec('myconn', 'SELECT another_stored_procedure()');
PERFORM dblink_exec('myconn', 'COMMIT');
but I got an error in runtime:
ERROR: statement returning results not allowed
CONTEXT: SQL statement "SELECT dblink_exec('myconn', 'select another_stored_procedure()')"
so execution fails, although I tried to get the desired result in different ways.
UPDATE 1:
I know that stored procedures in postgresql are transactional.
I'm using dblink for the autonomous transactions functionality to use it on the same server.
The matter is that the default level of transactions on my server
is "read commited" but sometimes I need to start transactions with another level, e.g. "serializable".
So I need to execute stored procedure in autonomous transaction with explicit transaction level specifying.
And as far as I know dblink allows that, but I failed to find any useful info about dblink or dblink_exec functions which are suitable for my situation.
I assume, you have connected with another PostgreSQL server at the other end.
You need to call the dblink() function to execute statements, which has result(s), and not dblink_exec(). (Even if your function on the other end has returns void -- in that case, you could get a single NULL from calling that function in a SELECT.)
Also, you might not need transaction management:
Are PostgreSQL functions transactional?
In short, you need to execute:
-- PERFORM dblink_exec('myconn', 'BEGIN ...');
-- if you need explicit transaction management
PERFORM * FROM dblink('myconn', 'SELECT another_stored_procedure()') alias(col text);
-- PERFORM dblink_exec('myconn', 'COMMIT');
I have a table called MyTable on which I have defined a trigger, like so:
CREATE TRIGGER dbo.trg_Ins_MyTable
ON dbo.MyTable
FOR INSERT
AS
BEGIN
SET NOCOUNT ON;
insert SomeLinkedSrv.Catalog.dbo.OtherTable
(MyTableId, IsProcessing, ModifiedOn)
values (-1, 0, GETUTCDATE())
END
GO
Whenever I try to insert a row in MyTable, I get this error message:
Msg 3910, Level 16, State 2, Line 1
Transaction context in use by another session.
I have SomeLinkedSrv properly defined as a linked server (for example, select * from SomeLinkedSrv.Catalog.dbo.OtherTable works just fine).
How can I avoid the error and successfully insert record+execute the trigger?
Loopback linked servers can't be used in a distributed transaction if MARS is enabled.
Loopback linked servers cannot be used in a distributed transaction.
Trying a distributed query against a loopback linked server from
within a distributed transaction causes an error, such as error 3910:
"[Microsoft][ODBC SQL Server Driver][SQL Server]Transaction context in
use by another session." This restriction does not apply when an
INSERT...EXECUTE statement, issued by a connection that does not have
multiple active result sets (MARS) enabled, executes against a
loopback linked server. Note that the restriction still applies when
MARS is enabled on a connection.
http://msdn.microsoft.com/en-us/library/ms188716(SQL.105).aspx
I solve It.
I was using the same linked server to call the second procedure and then into the procedure I was using the same linked server.
It's very Easy, only we have to know the restricctions of linked servers.
I have resolved it by removing linked server used in the stored procedure and then called stored procedure by the same linked server. It wasnt working in DEV environement.
One of causes of this situation is a trigger that works for linked-sever database table. An also SQL version of SQL-Server which processes database matters. To avoid this ERROR during sql query execution we should temporarily disable and after execution enable triggers for tables updated. All with database name check. Here is an example:
Select * From People where PersonId In (#PersonId, #PersonIdRight)
IF 'DOUBLE' = DB_NAME()
ALTER TABLE [dbo].[PeopleSites] DISABLE TRIGGER [PeopleSites_ENTDB_UPDATE]
Update PeopleSites Set PersonId = #PersonIdRight Where PersonId = #PersonId
IF 'DOUBLE' = DB_NAME()
ALTER TABLE [dbo].[PeopleSites] ENABLE TRIGGER [PeopleSites_ENTDB_UPDATE]
Select * From PeopleSites where PersonId In (#PersonId, #PersonIdRight)
I also got the same error in our DEV environemnt, moving the linked databases to another sql instance resolved the issue. In our production environment these databases are already on separate instances
In my case I was using SQL 2005 and got "transaction context in use by another session" when running Insert....exec over a linked server. The fix for me was to patch from SP2 build 3161 to SP3. SP2 cumulative 5 is supposed to fix though.
https://support.microsoft.com/en-us/kb/947486
When remote database sits on the same server,configure the linked server without specifying the database server ip / hostname and port. Just the database name should be sufficient.
I was getting the same "transaction context in use by another session error" when trying to run an UPDATE query:
BEGIN TRAN
--ROLLBACK TRAN
--COMMIT TRAN
UPDATE did
SET did.IsProcessed = 0,
did.ProcessingLockID = NULL
FROM [proddb\production].DLP.dbo.tbl_DLPID did (NOLOCK)
WHERE did.dlpid IN ('bunch of GUIDs')
--WHERE did.DLPID IN (SELECT DLPID FROM #TableWithData)
However I didn't realize I was already trying to run this on the DLP database on the ProdDb\Production server. Once I removed that "[proddb\production].DLP.dbo." prefix from the query, it worked fine.
I have a stored procedure that retrieves sensitive information from an SQL Server 2008 database. I would like to modify the procedure so that any time it is called, it records information about who called it in a separate table.
I thought something like the following would work:
declare #account varchar(255);
set #account = (SELECT SYSTEM_USER);
INSERT into AUDIT_LOG(ACCOUNT, TSTAMP)
VALUES(#account, getdate())
;
--Now fetch data
SELECT x,y,z from sensitive_info;
My issue is that the client application can issue a call to this stored procedure and get the sensitive information, but not commit the connection and the INSERT never occurs!
Is there some way to force the INSERT to happen before the SELECT?
I am using SQL Server 2008.
Thanks,
Carl
You only COMMIT if a transaction has been started.
So you can test for an open transaction first and disallow the read. This will ensure that no transaction is open to be rolled back. I've used XACT_STATE() here
Using SET XACT_ABORT ON and TRY/CATCH too will mean that the INSERT for logging must happen too before the read happens. Any errors at all on INSERT will go to the CATCH block. So no read and the logging fail can itself be logged too.
So: this is your guarantee of "read only if logged"
Having an explicit transaction doesn't help: the INSERT is an atomic action anyway. And if the called opens a transaction the log entry can be rolled back
CREATE PROC getSecretStuff
AS
SET NOCOUNT, XACT_ABORT ON;
BEGIN TRY
IF XACT_STATE() <> 0
RAISERRROR ('Call not allowed in an active transaction', 16, 1)
INSERT into AUDIT_LOG(ACCOUNT, TSTAMP)
VALUES(SYSTEM_USER, getdate());
--Now fetch data
SELECT x,y,z from sensitive_info;
END TRY
BEGIN CATCH
-- error handling etc
END CATCH
GO
Why not use the build in auditing functionality?
Have you tried using expicit transactions and doing the select after the commit statement?
On you insert a record in a table you should be albe to get the SCOPE_IDENTITY() of the ast inserted value. Before doing SELECT x,y,z from sensitive_info; you can check if SCOPE_IDENTITY() > 0 then only execute SELECT statement.
I'm trying to connect from Microsoft SQL server to as AS/400 so i can pull data from the AS/400 then flag the data as being pulled.
I've successfully created and OLE DB "IBMDASQL" connection, and am able to pull data some data, but i'm running into an issue when i try to pull data from a very large table
This runs fine, and returns a count of 170 million:
select count(*)
from transactions
This query executed for 15 hours before i gave up on it. (It should return zero since i haven't flagged anything as 'in process' yet)
select count(*)
from transactions
where processed = 'In process'
I'm a Microsoft guy, but my AS/400 guy says that there is an index on the 'processed' column and that locally, that query run instantaneously.
Any thoughts on what i might be doing wrong? I found a table with only 68 records in it, and was able to run this query in about a second:
select count(*)
from smallTable
where RandomColumn = 'randomValue'
So I know that the AS/400 is at least able to understand that type of query.
I have had to fight this battle many times.
There are two ways of approaching this.
1) Stage your data from the AS400 into SQL server where you can optimize your indexes
2) Ask the AS400 folks to create logical views which speed up data retrieval, your AS400 programmer is correct, index will help but I forget the term they use to define a "view" similar to a sql server view, I beleive its something like "physical" v/s "logical". Logical is what you want.
Thirdly, 170 million is a lot of records, even for a relational database like SQL server, have you considered running an SSIS package nightly that stages your data into your own SQL table to see if it improves performance?
I would suggest this way to have good performance, i suppose you have at least SQL2005, i havent tested yet but this is a tip
Let the AS400 perform the select in native way by creating stored procedure in the AS400
open a AS400 session
launch STRSQL
create an AS400 stored procedure in this way to get/update the recordset
CREATE PROCEDURE MYSELECT (IN PARAM CHAR(10))
LANGUAGE SQL
DYNAMIC RESULT SETS 1
BEGIN
DECLARE C1 CURSOR FOR SELECT * FROM MYLIB.MYFILE WHERE MYFIELD=PARAM;
OPEN C1;
RETURN;
END
create an AS400 stored procedure to update the recordset
CREATE PROCEDURE MYUPDATE (IN PARAM CHAR(10))
LANGUAGE SQL
RESULT SETS 0
BEGIN
UPDATE MYLIB.MYFILE SET MYFIELD='newvalue' WHERE MYFIELD=PARAM;
END
Call those AS400 SP from SQL SERVER
declare #myParam char(10)
set #myParam = 'In process'
-- get the recordset
EXEC ('CALL NAME_AS400.MYLIB.MYSELECT(?) ', #myParam) AT AS400 -- < AS400 = name of linked server
-- update
EXEC ('CALL NAME_AS400.MYLIB.MYUPDATE(?) ', #myParam) AT AS400
Hope it helps
I recommend following the suggestions in the IBM Redbook SQL Performance Diagnosis on IBM DB2 Universal Database for iSeries to determine what's really happening.
IBM technical support can also be extremely helpful in diagnosing issues such as these. Don't be afraid to get in touch with them as the software support is generally included as part of the maintenance contract and there is no charge to talk to them.
I've seen OLEDB connections eat up 100% cpu for hours and when the same query is run through VisualExplain (query analyzer) it estimates mere seconds to execute.
We found that running the query like this performed liked expected:
SELECT *
FROM OpenQuery( LinkedServer,
'select count(*)
from transactions
where processed = ''In process''')
GO
Could this be a collation problem? - your WHERE clause is testing on a text field and if the collations of the two servers don't match this clause will be applied clientside rather than serverside so you are first of all pulling all 170 million records down to the client and then performing the WHERE clause on it there.
Based on the past interactions I have had, the query should take about the same amount of time no matter how you access the data. Another thought would be if you could create a view on the table to get the data you need or use a stored procedure.