We have spent several weeks trying to fix an issue that occurs in the customer's production environment and does not occur in our test environment.
After several analyses, we have found that this error occurs only when one condition is met: processing times greater than 3600 seconds in the API.
The situation is the following:
SAP is connected to a server with Windows Server 2016 and IIS 10.0 where we have an API that is responsible for interacting with a DB use by an external system.
The process that we execute sends data from SAP to the API and this, with the data it receives from SAP and the data it obtains from the DB of the external system, performs a processing and a subsequent update in the DB.
This process finishes without problems when the processing time in the API is less than 3600 seconds.
On the other hand, when the processing time is greater than 3600 seconds, the API generates the response correctly, and the server tries to return the response to SAP, but it is not possible.
Below I show an example of a server log entry when it tries to return a response after more than 3600 seconds of API processing. As you can see, a 995 error occurs: (I have censored some parts)
Any idea where the error could come from?
We have compared IIS configurations in Production and Test. We have also reviewed the parameters of the SAP system in Production and Test and we have not found anything either.
I remain at your disposal to provide any type of additional information that may be useful for solving the problem.
UPDATE 1 - 02/09/2022
After enabling FRT (Failed Request Tracing) on IIS for 200 response codes, looking at the event log of the request that is causing the error, we have seen this event at the end:
Any information about what could be causing this error? ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)"
UPDATE 2 - 02/09/2022
Comparing configurations from customer's environment and our test environment:
There is a Firewall between SAP Server and IIS Server with the default idle timeout configured for TCP (3600 seconds). This is not happening in Test Environment because there is no Firewall.
Establishing a Firewall policy specifying a custom idle timeout for this service (7200 seconds) the problem will be solved.
sc-win32 status 995, the I/O operation has been aborted because of
either a thread exit or an application request.
Please check the setting of minBytesPerSecond configuration parameter in IIS. The default "minBytesPerSecond" is 240.
Specifies the minimum throughput rate, in bytes, that HTTP.sys
enforces when it sends a response to the client. The minBytesPerSecond
attribute prevents malicious or malfunctioning software clients from
using resources by holding a connection open with minimal data. If the
throughput rate is lower than the minBytesPerSecond setting, the
connection is terminated.
Related
I have an Azure function app. I am facing the Socket access permission issue.
The Issue:
We face a socket connection issue with the message. This message came when we execute HttpClient to the server. HttpClient does not return any other information with error message.
An attempt was made to access a socket in a way forbidden by its access permissions. (xyz.server.com:port)
This Issue is not happening every time, it happens randomly for some time of period, and then goes.
Reason for the Issue:
We neither have any idea about the cause of the issue nor are sure if the issue occurred at Azure Level due to some network setup or occurred from the server we are trying to connect to via HTTP Client.
Please help us with how we can check the Azure side to identify the issue. We already Informed the Support team of the other server where CRUD operation happens.
Azure Function App Config
Region: US-West 2
OS: Window
Function Runtime: 4.15.2.20177
Mode: Run from Precompiled Package
App Code Setting
Framework: NET 6.0
C#: 10
Ready To Run: true
Network
Inbound Access Restriction: Off
Inbound App Assigned Address: off
Outbound VNET: N/A
Outbound Hybrid Connection: N/A
Outbound NAT Gateway: N/A
Note:
This Stack overflow Thread mentions the same kind of issue. over there one answer suggests using HTTPWebRequest and Response instead of HttpClient.
That I will check in some time if it fixes the issue or not?
Process detail:
The Azure Function App is getting called from the scheduler from Our Server every X Min. When it's being executed.
It will check for a particular status let's say Start a Thread. The function App will start a new thread and change the status Thread is Running and change it again Thread completed once the task in the thread is completed.
This Thread created by Function uses HttpClient to connect with another server to CRUD operation for different data based on unique identifiers. (No of Identifier need to process will vary from time to time)
Let's say we have to process 200 identifiers. Now Thread is processing all identifiers one by one (we do that as we don't want to overwhelm our server where we doing) Async CRUD operation.
The deploy to some of the servers takes extremely long. Where it normally takes like 30 seconds to download the artifact on some servers it can take over 8 minutes, sometimes the deploy even fails if it takes too long. This behavior is consistent for the same server and it didn't changed for atleast the past 2 weeks.
Internet connectivity is good and I see 1gbit up and down with a speedtest on these servers. We use west europe as region and I can confirm this by looking at the urls in the log.
In the log I see these kind of messages:
2022-04-19T08:07:15.8187615Z ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/someguid/_apis/dedup/urls attempt 1/6 failed with TimeoutException: 'The HTTP request timed out after 00:01:40.'
What can cause this to happen?
I have a powershell script to check a webserver page availability and task runs for every 5mins.
But I am getting TLS/SSL request is aborted error for every 3-4 hour once.
I have used the below one in my code and it's works fine.
[Net.ServicePointManager]::SecurityProtocol =[Net.SecurityProtocolType]::Tls12
I had setup timeout secs(100) for invoking my web request but it doesn't seem to work this way.
Someone can help me why it's giving TLS/SSL errors all of a sudden? And how to fix this?
This is an interesting issue. It could be many issues:
This could be an issue with the server; Perhaps the server is dropping requests or has some internal handling issues.
The request may be timing out before it has a chance to receive a response.
There could be network errors unrelated to your script or the target server.
I wrap your web request method in a try-catch, if it catches than it tries again, up to a certain specified amount of times. If this does not fix it, there is a chance that the problem lies with the server and not your script.
I am sending a soap request using soapUI to fetch data from oracle 10g db. Soap UI successfully displays response when the db fetches results within 30 seconds.
But the real problem is when the db response exceeds 30 seconds, soapUI displays following error message :
Fault occurred while processing.
I have tried the below 3 scenarios :
1) Increased the socket timeout to 1200000
2) Increased the timeout values in tomcat server config file (/conf/server.xml)
3) Checked for any Null Pointer exception and found none.
Please help me to get success message in soapUI. Thanks in advance.
There are many components between the SoapUI adapter and the database engine. Most of these will have a configurable timeout.
The listener on the database server will pass the query to the database engine - and the database engine itself will have some protection against long-running queries. It's quite likely that the database is killing queries that run over 30 seconds.
You can prove this by capturing a query from your application and trying the same query directly in the database administration tool. This will tell you why the query fails (if it fails.)
JDBC calls a component listening on the database server - this flow itself will have a timeout, which you can set at the JDBC level somewhere in your environment.
I provide support for a large application across multiple servers. System has been running live for 6+ months.
8th December: total system failure. iisreset across each of the servers sorted it out. Everything back to normal.
Post failure investigation showed various processes not able to get a response from a particular server which hosts an instance of Dynamics CRM (2011 R11). Specifically it seems the SOAP service was not responding (Organization.svc). 503 - Server Unavailable (really it was just the web service). I suspect it died.
Having the exact time of the error I checked the event logs on the server but these did not have anything of use. The last error prior to the failure was a report rendering error which was 9 minutes before the system actually went down. Surely if web service crashed this would be reflected in the event log?
Fast forward to today, 8th January and the system fails again. The 8th of the month again! iisreset fixes it... again!
Again, completely useless event logs showings no errors prior to failure.
Entertained the idea of Dynamics CRM trace logging but this is out of the question due to the performance hit.
Apart from the event logs where else to look? Are there possible external factors or causes? I'm trying to find the root cause but have run out of ideas!
While this may not address the source of your problem, maybe it can help minimize the symptoms. May I suggest that you configure the IIS server to recycle the application pool at a scheduled interval within your production environment.
http://technet.microsoft.com/en-us/library/cc753179%28v=ws.10%29.aspx