I cannot seem to extend the 110 second timeout for requests to my Azure Web App. I have done the following in order to increase this limit, but with no success.
ASP.NET's HTTP request execution timeout (web.config):
<system.web>
<httpRuntime executionTimeout="600" />
</system.web>
IIS connection timeout (web.config):
<system.applicationHost>
<webLimits connectionTimeout="00:10:00" />
</system.applicationHost>
Kudu timeout before external commands are killed (site app setting):
SCM_COMMAND_IDLE_TIMEOUT = 600
What am I missing?
request timeout of 110s
It is very odd that your request timeout of 110s,From the Auzre official document, we could know that default timeout is about 4 minutes.It seems that we are not able to increase request timeout. The following is the snippet from the document. Please have a try to scale up and scale down back the App service plan. If still have the same issue, please connect to Azure support team for more help.
Azure Load Balancer has a default idle timeout setting of four minutes. This is generally a reasonable response time limit for a web request. If your web app requires background processing, we recommend using Azure WebJobs. The Azure web app can call WebJobs and be notified when background processing is finished. You can choose from multiple methods for using WebJobs, including queues and triggers.
+
WebJobs is designed for background processing. You can do as much background processing as you want in a WebJob. For more information about WebJobs, see Run background tasks with WebJobs.
Note :SCM_COMMAND_IDLE_TIMEOUT = 600 could use for your build process launches some command in the server side. But the request is timeout that will cause clients to get disconnected after 230 seconds, we could get more info from Azure Kudu Configurable settings.
Related
I have an Azure function app. I am facing the Socket access permission issue.
The Issue:
We face a socket connection issue with the message. This message came when we execute HttpClient to the server. HttpClient does not return any other information with error message.
An attempt was made to access a socket in a way forbidden by its access permissions. (xyz.server.com:port)
This Issue is not happening every time, it happens randomly for some time of period, and then goes.
Reason for the Issue:
We neither have any idea about the cause of the issue nor are sure if the issue occurred at Azure Level due to some network setup or occurred from the server we are trying to connect to via HTTP Client.
Please help us with how we can check the Azure side to identify the issue. We already Informed the Support team of the other server where CRUD operation happens.
Azure Function App Config
Region: US-West 2
OS: Window
Function Runtime: 4.15.2.20177
Mode: Run from Precompiled Package
App Code Setting
Framework: NET 6.0
C#: 10
Ready To Run: true
Network
Inbound Access Restriction: Off
Inbound App Assigned Address: off
Outbound VNET: N/A
Outbound Hybrid Connection: N/A
Outbound NAT Gateway: N/A
Note:
This Stack overflow Thread mentions the same kind of issue. over there one answer suggests using HTTPWebRequest and Response instead of HttpClient.
That I will check in some time if it fixes the issue or not?
Process detail:
The Azure Function App is getting called from the scheduler from Our Server every X Min. When it's being executed.
It will check for a particular status let's say Start a Thread. The function App will start a new thread and change the status Thread is Running and change it again Thread completed once the task in the thread is completed.
This Thread created by Function uses HttpClient to connect with another server to CRUD operation for different data based on unique identifiers. (No of Identifier need to process will vary from time to time)
Let's say we have to process 200 identifiers. Now Thread is processing all identifiers one by one (we do that as we don't want to overwhelm our server where we doing) Async CRUD operation.
We have spent several weeks trying to fix an issue that occurs in the customer's production environment and does not occur in our test environment.
After several analyses, we have found that this error occurs only when one condition is met: processing times greater than 3600 seconds in the API.
The situation is the following:
SAP is connected to a server with Windows Server 2016 and IIS 10.0 where we have an API that is responsible for interacting with a DB use by an external system.
The process that we execute sends data from SAP to the API and this, with the data it receives from SAP and the data it obtains from the DB of the external system, performs a processing and a subsequent update in the DB.
This process finishes without problems when the processing time in the API is less than 3600 seconds.
On the other hand, when the processing time is greater than 3600 seconds, the API generates the response correctly, and the server tries to return the response to SAP, but it is not possible.
Below I show an example of a server log entry when it tries to return a response after more than 3600 seconds of API processing. As you can see, a 995 error occurs: (I have censored some parts)
Any idea where the error could come from?
We have compared IIS configurations in Production and Test. We have also reviewed the parameters of the SAP system in Production and Test and we have not found anything either.
I remain at your disposal to provide any type of additional information that may be useful for solving the problem.
UPDATE 1 - 02/09/2022
After enabling FRT (Failed Request Tracing) on IIS for 200 response codes, looking at the event log of the request that is causing the error, we have seen this event at the end:
Any information about what could be causing this error? ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)"
UPDATE 2 - 02/09/2022
Comparing configurations from customer's environment and our test environment:
There is a Firewall between SAP Server and IIS Server with the default idle timeout configured for TCP (3600 seconds). This is not happening in Test Environment because there is no Firewall.
Establishing a Firewall policy specifying a custom idle timeout for this service (7200 seconds) the problem will be solved.
sc-win32 status 995, the I/O operation has been aborted because of
either a thread exit or an application request.
Please check the setting of minBytesPerSecond configuration parameter in IIS. The default "minBytesPerSecond" is 240.
Specifies the minimum throughput rate, in bytes, that HTTP.sys
enforces when it sends a response to the client. The minBytesPerSecond
attribute prevents malicious or malfunctioning software clients from
using resources by holding a connection open with minimal data. If the
throughput rate is lower than the minBytesPerSecond setting, the
connection is terminated.
In our dev environment we have lots of repos, lots of builds and lots of buildservers, and most of the time things work just like they should - however, we are seeing an increase in builds that fail because of timeouts.
These timeouts are not happening because we are getting close to the limit, but because something "gets stuck/blocked" in the pipeline and it stays on that step until timeout kills the build.
To better debug why that happens, we need to be able to query what builds fails because of this timeout, so we for instance can see, if it is a particular build server or agent that has this problem.
We can not find anything in the API that would give us the timeout error, but we can see that the UI is able to deduct it somehow:
So far we have narrowed it down to query all builds with completed status (through this API), but we get no completion reason, and buildtimes are never exact the same as the timeout of the build defintion, so "guessing" it from the execution plan will also be a bit shaky.
How can we filter our builds down to only the builds that have timed out?
We can use the below API to get details for a build.
Note: do not add timelineId, we should list all info
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline?api-version=6.1-preview.2
If the build is canceled because of the timeout setting, we can get the message: The job running on agent Hosted Agent ran longer than the maximum time of xxx minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
By the way, we can use the API Builds - List to filter all failed build. if the build is canceled due to a timeout setting. the result is failed instead of cancel.
I have a job which uses tRestClient to get a token, then uses tRest to get data from CRM based on the token. Both of these components seem to timeout sometimes while connecting to CRM. How can I set them up to have 10 retries for the connection and more minutes to wait before they timeout?
There is some setting to do that.
Check the documentation here:
https://help.talend.com/reader/7NvFnkWpbH8Gy3Rm6mUXnw/ECeCwoP1aVopmhqJe_dENA.
You need to go in the advanced setting in the component.
Moreover, there is some limitation to the studio like you see in the documentation.
After creating a subscription in the Reporting Configuration, the emails are not sent at the specified time, but if you perform any actions on the site, the emails will be sent. This is the problem of the virtual machine on which the site or IIS is deployed. Or does the feature Kentico.
Prompt in what there can be a problem and how the system of sending of emails inin the Reporting Configuration works.
This is a correct behavior. Kentico checks for e-mails that are to be sent at the end of requests which means that if there are no requests, no e-mails are sent. If you need some tasks to be send at a specified time, you need to use Windows scheduler and configure task (e-mmail sending) to use it. See official documentation for more detail.s
Another option to Enn's, is to use a service like UpTime Robot to continually visit your page (like every 5 minutes), this not only generates a request but also will help keep your site awake and from going to sleep if you can't manually set the Worker Processor to never go to sleep.
Windows Scheduler is the most reliable, but UpTime robot has worked well for us.
https://uptimerobot.com/
Under the "Scheduled Tasks" module there is a schedule task called "Report subscription sender". This task runs every minute and is responsible for checking your subscriptions and sending e-mails based on your configuration. This task runs within the Kentico instance. By default IIS will put the site/app pool to sleep after x (default 20) minutes of idle and the scheduled tasks are no longer able to run. When you hit the site it wakes the process back up and the scheduled task is able to run again. You can go into IIS and configure the "Idle Time-out (minutes)" for the application pool see this link https://patrickdesjardins.com/blog/iis-no-sleep-idle-and-autostart-with-values for a pretty good illustration. You can also adjust the app pool recycle intervals but that is probably not necessary for your issue.
The other option, as mentioned by Enn, is to install the Kentico Scheduler Windows service which always runs and configure that scheduled task to run in the service.