HttpClient Error: `An attempt was made to access a socket in a way forbidden by its access permission` from Azure Function Thread - sockets

I have an Azure function app. I am facing the Socket access permission issue.
The Issue:
We face a socket connection issue with the message. This message came when we execute HttpClient to the server. HttpClient does not return any other information with error message.
An attempt was made to access a socket in a way forbidden by its access permissions. (xyz.server.com:port)
This Issue is not happening every time, it happens randomly for some time of period, and then goes.
Reason for the Issue:
We neither have any idea about the cause of the issue nor are sure if the issue occurred at Azure Level due to some network setup or occurred from the server we are trying to connect to via HTTP Client.
Please help us with how we can check the Azure side to identify the issue. We already Informed the Support team of the other server where CRUD operation happens.
Azure Function App Config
Region: US-West 2
OS: Window
Function Runtime: 4.15.2.20177
Mode: Run from Precompiled Package
App Code Setting
Framework: NET 6.0
C#: 10
Ready To Run: true
Network
Inbound Access Restriction: Off
Inbound App Assigned Address: off
Outbound VNET: N/A
Outbound Hybrid Connection: N/A
Outbound NAT Gateway: N/A
Note:
This Stack overflow Thread mentions the same kind of issue. over there one answer suggests using HTTPWebRequest and Response instead of HttpClient.
That I will check in some time if it fixes the issue or not?
Process detail:
The Azure Function App is getting called from the scheduler from Our Server every X Min. When it's being executed.
It will check for a particular status let's say Start a Thread. The function App will start a new thread and change the status Thread is Running and change it again Thread completed once the task in the thread is completed.
This Thread created by Function uses HttpClient to connect with another server to CRUD operation for different data based on unique identifiers. (No of Identifier need to process will vary from time to time)
Let's say we have to process 200 identifiers. Now Thread is processing all identifiers one by one (we do that as we don't want to overwhelm our server where we doing) Async CRUD operation.

Related

The listener for Azure function was unable to start - Microsoft.Azure.EventHubs.Processor Encountered error

I am getting the below error whilst running my Python Azure Function on the local machine in VSCode.
For clarification the message is:
The listener for function 'Functions.IoT_Data-Handler' was unable to
start. Microsoft.Azure.EventHubs.Processor: Encountered error while
fetching the list of EventHub PartitionIds. System.Private.CoreLib: A
connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed
because connected host has failed to respond.
This error has never occurred before in the time I have started using VSCode for Azure functions (since last September). The only thing that has changed recently is that I now deploy this function within an Azure Function premium resource, but really that should not matter in the dev environment.
For information, this function is hooked up to an Azure IoT-Hub endpoint and is simply reading and processing the uplink data before saving it to an Azure SQL database.
Can anyone offer any advice?
Check if my below findings help to fix your issue:
As #PeterBons said, check the connection string given correctly in the local.settings.json:
Whatever the Event Hub Endpoint/IoT Hub Endpoint Connection String given in the file local.settings.json, that property name should be mapped in the function.json file.
Try replacing the IoT Hub Connection String without the consumer group name as mentioned in this GitHub Issue #5512
I found similar issues in the SO 1 & 2 which will be helpful to fix your issue.

REST API does not return answer back after more than 3600 seconds of processing

We have spent several weeks trying to fix an issue that occurs in the customer's production environment and does not occur in our test environment.
After several analyses, we have found that this error occurs only when one condition is met: processing times greater than 3600 seconds in the API.
The situation is the following:
SAP is connected to a server with Windows Server 2016 and IIS 10.0 where we have an API that is responsible for interacting with a DB use by an external system.
The process that we execute sends data from SAP to the API and this, with the data it receives from SAP and the data it obtains from the DB of the external system, performs a processing and a subsequent update in the DB.
This process finishes without problems when the processing time in the API is less than 3600 seconds.
On the other hand, when the processing time is greater than 3600 seconds, the API generates the response correctly, and the server tries to return the response to SAP, but it is not possible.
Below I show an example of a server log entry when it tries to return a response after more than 3600 seconds of API processing. As you can see, a 995 error occurs: (I have censored some parts)
Any idea where the error could come from?
We have compared IIS configurations in Production and Test. We have also reviewed the parameters of the SAP system in Production and Test and we have not found anything either.
I remain at your disposal to provide any type of additional information that may be useful for solving the problem.
UPDATE 1 - 02/09/2022
After enabling FRT (Failed Request Tracing) on IIS for 200 response codes, looking at the event log of the request that is causing the error, we have seen this event at the end:
Any information about what could be causing this error? ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)"
UPDATE 2 - 02/09/2022
Comparing configurations from customer's environment and our test environment:
There is a Firewall between SAP Server and IIS Server with the default idle timeout configured for TCP (3600 seconds). This is not happening in Test Environment because there is no Firewall.
Establishing a Firewall policy specifying a custom idle timeout for this service (7200 seconds) the problem will be solved.
sc-win32 status 995, the I/O operation has been aborted because of
either a thread exit or an application request.
Please check the setting of minBytesPerSecond configuration parameter in IIS. The default "minBytesPerSecond" is 240.
Specifies the minimum throughput rate, in bytes, that HTTP.sys
enforces when it sends a response to the client. The minBytesPerSecond
attribute prevents malicious or malfunctioning software clients from
using resources by holding a connection open with minimal data. If the
throughput rate is lower than the minBytesPerSecond setting, the
connection is terminated.

Challenge in data from REST API using Azure Data Factory - access issue

We are trying to reach to an API hosted in our company network using rest connector in ADF (SHIR is used). Linked service connection is successful but dataset is unable to read the data and copy activity is as well failing with below error. Please suggest your thoughts in resolving the same.
Failure happened on 'Source' side. ErrorCode=UserErrorFailToReadFromRestResource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred while sending the request.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ,Source=System,'
This error is mostly seen due to firewall issues. You might want to verify your network firewall setting to allow the API request to be read.
Also, verify if your API call is working as expected using other API testing tools. If the issue persists you can raise a support ticket for engineers to investigate more on the issue.
If you are able to preview data in your source , then check your sink connection as this issue can occur when the Sink in the copy activity is behind a firewall, I was getting the same issue and I tried copying to a container without a firewall and it worked. Its weird that the error is related to Source and the issue is with Sink.

TLS/SSL error in between of web service checks

I have a powershell script to check a webserver page availability and task runs for every 5mins.
But I am getting TLS/SSL request is aborted error for every 3-4 hour once.
I have used the below one in my code and it's works fine.
[Net.ServicePointManager]::SecurityProtocol =[Net.SecurityProtocolType]::Tls12
I had setup timeout secs(100) for invoking my web request but it doesn't seem to work this way.
Someone can help me why it's giving TLS/SSL errors all of a sudden? And how to fix this?
This is an interesting issue. It could be many issues:
This could be an issue with the server; Perhaps the server is dropping requests or has some internal handling issues.
The request may be timing out before it has a chance to receive a response.
There could be network errors unrelated to your script or the target server.
I wrap your web request method in a try-catch, if it catches than it tries again, up to a certain specified amount of times. If this does not fix it, there is a chance that the problem lies with the server and not your script.

Could not open JDBC Connection, Unable to get managed connection for java during load test

Noticed below error during load test with multiple users and not in case of single SOAP request.
Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:
This could be due to any of the following:
The datasource connection pool has not been tuned (e.g. max-pool-size and blocking-timeout-millis) correctly for the maximum load on the application.
The application is leaking connections because it is not closing them and thereby returning them to the pool.
Threads with connections to the database are hanging and holding on to the connections.
Make sure that the min-pool-size and max-pool-size values for the respective datasource are set according to application load testing and connections are getting closed after use inside the application code.
Most likely you've found the bottleneck in your application, it seems that it cannot handle that many virtual users. The easiest solution would be raising an issue in your bug tracker system and let developers investigate it.
If you need to provide the root cause of the failure I can think of at least 2 reasons for this:
Your application or application server configuration is not suitable for high loads (i.e. number of connections in your JBOSS JDBC Connection pool configuration is lower than it is required given the number of virtual users you're simulating. Try amending min-pool-size and max-pool-size values to match the number of virtual users
Your database is overloaded hence cannot accept that many queries. In this case you can consider load testing the database separately (i.e. fire requests to the database directly via JMeter's JDBC Request sampler without hitting the SOAP endpoint of your application.) See The Real Secret to Building a Database Test Plan With JMeter article to learn more about database load testing concept.