We are currently trying out the SQLBulkCopy API on the new SQL Azure CTP.
While we have been able to consistently migrate tables with about a million rows, we are facing connection errors when working with larger tables. We keep getting (after random row transfers) the following error:
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
I understand that SQL Azure connection policies (mentioned here) state that the connection can be terminated for a number of reasons and it also mentions some error codes that are returned.But I am not able to understand which of these might be causing the error or capture the error code.
Is there a way we can get past this error and continue with the migration of table rows?
The SQLBulkCopy options used are:
BatchSize=1000
BulkCopyTimeout = 5000
Knowledge Base article 977291 gives this error message as a symptom of a Windows 2003 TCP/IP issue.
Related
I am getting the below error whilst running my Python Azure Function on the local machine in VSCode.
For clarification the message is:
The listener for function 'Functions.IoT_Data-Handler' was unable to
start. Microsoft.Azure.EventHubs.Processor: Encountered error while
fetching the list of EventHub PartitionIds. System.Private.CoreLib: A
connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed
because connected host has failed to respond.
This error has never occurred before in the time I have started using VSCode for Azure functions (since last September). The only thing that has changed recently is that I now deploy this function within an Azure Function premium resource, but really that should not matter in the dev environment.
For information, this function is hooked up to an Azure IoT-Hub endpoint and is simply reading and processing the uplink data before saving it to an Azure SQL database.
Can anyone offer any advice?
Check if my below findings help to fix your issue:
As #PeterBons said, check the connection string given correctly in the local.settings.json:
Whatever the Event Hub Endpoint/IoT Hub Endpoint Connection String given in the file local.settings.json, that property name should be mapped in the function.json file.
Try replacing the IoT Hub Connection String without the consumer group name as mentioned in this GitHub Issue #5512
I found similar issues in the SO 1 & 2 which will be helpful to fix your issue.
We are trying to reach to an API hosted in our company network using rest connector in ADF (SHIR is used). Linked service connection is successful but dataset is unable to read the data and copy activity is as well failing with below error. Please suggest your thoughts in resolving the same.
Failure happened on 'Source' side. ErrorCode=UserErrorFailToReadFromRestResource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred while sending the request.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ,Source=System,'
This error is mostly seen due to firewall issues. You might want to verify your network firewall setting to allow the API request to be read.
Also, verify if your API call is working as expected using other API testing tools. If the issue persists you can raise a support ticket for engineers to investigate more on the issue.
If you are able to preview data in your source , then check your sink connection as this issue can occur when the Sink in the copy activity is behind a firewall, I was getting the same issue and I tried copying to a container without a firewall and it worked. Its weird that the error is related to Source and the issue is with Sink.
I'm working with ADF and Azure Managed Postgres. I've had a reoccurring issue with look-ups and query-sourced copy activities timing out after about 35 seconds.
Failure happened on 'Source' side. 'Type=Npgsql.NpgsqlException,Message=Exception while reading from stream,Source=Npgsql,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond,Source=System,'
So the error says it's a Npgsql Exception, so I took a look at their documentation and modified the connection string to take Timeout = 60 and CommandTimeout = 60 as well (Internal Timeout will default to CommandTimeout).
And the queries still timeout at ~35 seconds. Could this be a socket issue with the Azure Managed Instance causing the timeout and it's just propagating down to npgsql?
Any help would be appreciated!
I just want to add some precision because I had the same problem (and thanks #DeliciousMalware and #Leon_Yue):
There is a default timeout of 30s for requests with a postgres connection
There is no way to change this timeout from the lookup activity directly.
The only option that does something is to add Timeout=600;CommandTimeout=0; to your connection string in your linked service (if you use a key vault for exemple) or add the options in the linked service additionnal parameters like in #DeliciousMalware screenshot.
Timeout is to establish the connection, and CommandTimeout is the timeout for the command itself (in second, 0 means infinity)
The library behind the connection is npgsql, and the others parameters and details that are usable are there: https://www.npgsql.org/doc/connection-string-parameters.html
I had a hard time to find what the parameters of the connection string are and what they mean, and which one exists, so I was really happy to find this doc. I didn't found a lot of doc on postgres in azure, so I though this list of param would be of some use for others.
I added the 2 parameters suggested by Leon and that resolved the issue I had.
Here is a screenshot of the parameters being added to the linked service:
Here is a screenshot of the error and completed run:
Here is a screenshot of the error and completed run:
Problem description:
Resource and computational intensive ETL pipeline job running every minute are failing. We receive a lot of same error message as shown below. (Seems like its failing almost every minute now). We did observe that the new parameter template is based on a generic template and can be optimized. For example work_mem is 4MB and we increased it. Any advise would be appreciated Thanks in advance.
Steps we have taken to resolve the issue:
vacuum analyze
increased work mem
Error Message 1:
(OperationalError('SSL SYSCALL error: EOF detected\n')
ETL SQL (upsert, update, create temp table, indexing, sorting, etc)
....
Exception: connection already closed
Error Message 2:
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
My problem is pretty much the same as here
DB2 deadlock timeout Sqlstate: 40001, reason code 68 due to update statements called from servlet using SQL
The problem is that I am using dashDB which runs as a service in IBM Cloud (formerly known as bluemix), so I don't have access to the same administrative tools some DB2 DBA has access to (AFAIK).
So I have a simple table, but when I try to add a column, I get this error
SQL0911N: SQL0911N The current transaction has been rolled back
because of a deadlock or timeout. Reason code "68". SQLSTATE=40001
[IBM][CLI Driver][DB2/LINUXX8664] SQL0911N The current transaction has
been rolled back because of a deadlock or timeout. Reason code "68".
SQLSTATE=40001
I've stopped all other DB activity such as other select statements, and I've tried using eclipse JDBC-based DB IDE instead of the web-based administrative DashDB provided by IBM Cloud (just because its authentication session just ends too quickly) without success.
Try this and post if it works.
http://www-01.ibm.com/support/docview.wss?uid=swg21440972