The deploy to some of the servers takes extremely long. Where it normally takes like 30 seconds to download the artifact on some servers it can take over 8 minutes, sometimes the deploy even fails if it takes too long. This behavior is consistent for the same server and it didn't changed for atleast the past 2 weeks.
Internet connectivity is good and I see 1gbit up and down with a speedtest on these servers. We use west europe as region and I can confirm this by looking at the urls in the log.
In the log I see these kind of messages:
2022-04-19T08:07:15.8187615Z ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/someguid/_apis/dedup/urls attempt 1/6 failed with TimeoutException: 'The HTTP request timed out after 00:01:40.'
What can cause this to happen?
Related
We have spent several weeks trying to fix an issue that occurs in the customer's production environment and does not occur in our test environment.
After several analyses, we have found that this error occurs only when one condition is met: processing times greater than 3600 seconds in the API.
The situation is the following:
SAP is connected to a server with Windows Server 2016 and IIS 10.0 where we have an API that is responsible for interacting with a DB use by an external system.
The process that we execute sends data from SAP to the API and this, with the data it receives from SAP and the data it obtains from the DB of the external system, performs a processing and a subsequent update in the DB.
This process finishes without problems when the processing time in the API is less than 3600 seconds.
On the other hand, when the processing time is greater than 3600 seconds, the API generates the response correctly, and the server tries to return the response to SAP, but it is not possible.
Below I show an example of a server log entry when it tries to return a response after more than 3600 seconds of API processing. As you can see, a 995 error occurs: (I have censored some parts)
Any idea where the error could come from?
We have compared IIS configurations in Production and Test. We have also reviewed the parameters of the SAP system in Production and Test and we have not found anything either.
I remain at your disposal to provide any type of additional information that may be useful for solving the problem.
UPDATE 1 - 02/09/2022
After enabling FRT (Failed Request Tracing) on IIS for 200 response codes, looking at the event log of the request that is causing the error, we have seen this event at the end:
Any information about what could be causing this error? ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)"
UPDATE 2 - 02/09/2022
Comparing configurations from customer's environment and our test environment:
There is a Firewall between SAP Server and IIS Server with the default idle timeout configured for TCP (3600 seconds). This is not happening in Test Environment because there is no Firewall.
Establishing a Firewall policy specifying a custom idle timeout for this service (7200 seconds) the problem will be solved.
sc-win32 status 995, the I/O operation has been aborted because of
either a thread exit or an application request.
Please check the setting of minBytesPerSecond configuration parameter in IIS. The default "minBytesPerSecond" is 240.
Specifies the minimum throughput rate, in bytes, that HTTP.sys
enforces when it sends a response to the client. The minBytesPerSecond
attribute prevents malicious or malfunctioning software clients from
using resources by holding a connection open with minimal data. If the
throughput rate is lower than the minBytesPerSecond setting, the
connection is terminated.
I have a Nuxt.js server side website deployed on Vercel. I've noticed that, on some occasions, I get a 504: GATEWAY_TIMEOUT error, with the code FUNCTION_INVOCATION_TIMEOUT and the message "This Serverless Function has timed out".
Why am I seeing this error?
Vercel imposes some limits when using their platform. This includes a serverless function execution timeout, which is basically the amount of time that a serverless function is allowed to process an HTTP request before it must respond.
Recently, they have silently reduced the serverless function execution timeout from 10s for the hobby plan, 60s for the pro plan and 1000s for the enterprise plan to, respectively, 5s, 15s and 30s. This might be the reason why you're experiencing this error now.
I really didn't like the fact that they lowered these limits without warning their users (I wasn't warned, at least) and giving them time to adjust.
As of 7 September 2022, the Hobby plan on Vercel supports Serverless Function Execution Timeout for 10 seconds. Pro for 60 seconds. Enterprise for 900 seconds.
Here is a link to their pricing plans: https://vercel.com/pricing
Application Error
The request for a Serverless Function reached the timeout threshold (5 seconds for Hobby accounts, 15 seconds for Pro accounts, and 30 seconds for Enterprise accounts). This is an application error.
Getting worse ...
I'm using Powershell with resource templates to provision SQL Servers (and databases) in Azure. After a few provisions it starts returning this error:
New-AzureRmResourceGroupDeployment : 2:40:18 PM - Resource Microsoft.Sql/servers 'oao-sql01-gd6helghx' failed with message 'Subscription 'mysubscription Guid here'
is not ready for the operation because another operation is currently in progress. Please wait a few minutes and then try the operation again.'
There is no operation currently in progress that I can ascertain. Any Sql Servers I have provisioned are operational and I am able to provision other kinds of resources with no problems.
If I wait several hours (not a few minutes), it will let me provision a new server but soon after blocks me again. Is there some kind of throttling going on if I am doing a bunch of provisioning/teardown?
Any ideas are appreciated.
Turns out that after Azure support bumped my quota of SQL Server, this began to work a while later. Not sure why it took so long to kick in as I only had 1 server at the time. The best explanation was that the counter tracking the number of DB's against our quota got stale and didn't clear for a while. The default quota is 6 SQL Server instances.
I posted on friday regarding this issue and received no response, however since then some updates have occurred which might affect whether people have an answer to my question or not.
I'm trying to deploy an MVC website to Azure, and in this particular project the web deploy receives a Socket Error 10054 and gives up after 10 attempts saying it was Unable to write data to the transport connection. It makes varying progress in between the socket failures but never completes within the 10 attempts.
I have since had a play around with other projects which are deployed to different url locations within this same Azure Account and they deploy fine! I think this means the problem is not on my end, i.e. port 8172 is open and deployment can be achieved with my current local settings.
What are the problems that can cause this socket error 10054? I saw somewhere that I should enable the "Allow untrusted certificate" option when deploying but I can't find that option within VS 2013.
Any ideas are welcome please,
This issue is driving me mad, it seems there's a real mixed bag of solution ideas which have worked for others but not me.
JK
I provide support for a large application across multiple servers. System has been running live for 6+ months.
8th December: total system failure. iisreset across each of the servers sorted it out. Everything back to normal.
Post failure investigation showed various processes not able to get a response from a particular server which hosts an instance of Dynamics CRM (2011 R11). Specifically it seems the SOAP service was not responding (Organization.svc). 503 - Server Unavailable (really it was just the web service). I suspect it died.
Having the exact time of the error I checked the event logs on the server but these did not have anything of use. The last error prior to the failure was a report rendering error which was 9 minutes before the system actually went down. Surely if web service crashed this would be reflected in the event log?
Fast forward to today, 8th January and the system fails again. The 8th of the month again! iisreset fixes it... again!
Again, completely useless event logs showings no errors prior to failure.
Entertained the idea of Dynamics CRM trace logging but this is out of the question due to the performance hit.
Apart from the event logs where else to look? Are there possible external factors or causes? I'm trying to find the root cause but have run out of ideas!
While this may not address the source of your problem, maybe it can help minimize the symptoms. May I suggest that you configure the IIS server to recycle the application pool at a scheduled interval within your production environment.
http://technet.microsoft.com/en-us/library/cc753179%28v=ws.10%29.aspx