Temporarily Unavailable GENERIC_HTTP_503 - rest

i have an application that runs through VM and performing multiple activities on it sometimes displays "Temporarily unavailable" message.
i have seen the debug logs and jetty server logs and it displays GENERIC_HTTP_503 error but for a rest call which is not known.
is there any way to know why exactly this problem is occurring inconsistently?
and is this because of server is overloaded with multiple requests?

Related

HttpClient Error: `An attempt was made to access a socket in a way forbidden by its access permission` from Azure Function Thread

I have an Azure function app. I am facing the Socket access permission issue.
The Issue:
We face a socket connection issue with the message. This message came when we execute HttpClient to the server. HttpClient does not return any other information with error message.
An attempt was made to access a socket in a way forbidden by its access permissions. (xyz.server.com:port)
This Issue is not happening every time, it happens randomly for some time of period, and then goes.
Reason for the Issue:
We neither have any idea about the cause of the issue nor are sure if the issue occurred at Azure Level due to some network setup or occurred from the server we are trying to connect to via HTTP Client.
Please help us with how we can check the Azure side to identify the issue. We already Informed the Support team of the other server where CRUD operation happens.
Azure Function App Config
Region: US-West 2
OS: Window
Function Runtime: 4.15.2.20177
Mode: Run from Precompiled Package
App Code Setting
Framework: NET 6.0
C#: 10
Ready To Run: true
Network
Inbound Access Restriction: Off
Inbound App Assigned Address: off
Outbound VNET: N/A
Outbound Hybrid Connection: N/A
Outbound NAT Gateway: N/A
Note:
This Stack overflow Thread mentions the same kind of issue. over there one answer suggests using HTTPWebRequest and Response instead of HttpClient.
That I will check in some time if it fixes the issue or not?
Process detail:
The Azure Function App is getting called from the scheduler from Our Server every X Min. When it's being executed.
It will check for a particular status let's say Start a Thread. The function App will start a new thread and change the status Thread is Running and change it again Thread completed once the task in the thread is completed.
This Thread created by Function uses HttpClient to connect with another server to CRUD operation for different data based on unique identifiers. (No of Identifier need to process will vary from time to time)
Let's say we have to process 200 identifiers. Now Thread is processing all identifiers one by one (we do that as we don't want to overwhelm our server where we doing) Async CRUD operation.

TLS/SSL error in between of web service checks

I have a powershell script to check a webserver page availability and task runs for every 5mins.
But I am getting TLS/SSL request is aborted error for every 3-4 hour once.
I have used the below one in my code and it's works fine.
[Net.ServicePointManager]::SecurityProtocol =[Net.SecurityProtocolType]::Tls12
I had setup timeout secs(100) for invoking my web request but it doesn't seem to work this way.
Someone can help me why it's giving TLS/SSL errors all of a sudden? And how to fix this?
This is an interesting issue. It could be many issues:
This could be an issue with the server; Perhaps the server is dropping requests or has some internal handling issues.
The request may be timing out before it has a chance to receive a response.
There could be network errors unrelated to your script or the target server.
I wrap your web request method in a try-catch, if it catches than it tries again, up to a certain specified amount of times. If this does not fix it, there is a chance that the problem lies with the server and not your script.

503 Server Unavailable - Dynamics CRM Web Service down - how to diagnose?

I provide support for a large application across multiple servers. System has been running live for 6+ months.
8th December: total system failure. iisreset across each of the servers sorted it out. Everything back to normal.
Post failure investigation showed various processes not able to get a response from a particular server which hosts an instance of Dynamics CRM (2011 R11). Specifically it seems the SOAP service was not responding (Organization.svc). 503 - Server Unavailable (really it was just the web service). I suspect it died.
Having the exact time of the error I checked the event logs on the server but these did not have anything of use. The last error prior to the failure was a report rendering error which was 9 minutes before the system actually went down. Surely if web service crashed this would be reflected in the event log?
Fast forward to today, 8th January and the system fails again. The 8th of the month again! iisreset fixes it... again!
Again, completely useless event logs showings no errors prior to failure.
Entertained the idea of Dynamics CRM trace logging but this is out of the question due to the performance hit.
Apart from the event logs where else to look? Are there possible external factors or causes? I'm trying to find the root cause but have run out of ideas!
While this may not address the source of your problem, maybe it can help minimize the symptoms. May I suggest that you configure the IIS server to recycle the application pool at a scheduled interval within your production environment.
http://technet.microsoft.com/en-us/library/cc753179%28v=ws.10%29.aspx

SQL Server Reporting Services (MSSQLSERVER) is screw up

I’m trying to investigate for which reason sometime my service SQL Server Reporting Services (MSSQLSERVER) is screwed up and doesn’t respond, producing an error in my SSRS reports. When the users run their report they get an ambiguous and confusing error message like: Server Error in ‘/Reports’ application.
This problem was resolved only by stopping and restarting the service. It comes only a few times, but I don't want to get it again.
The stability of the reporting is the first priority for our team. I want to know the reason or how can we prevent this error in the future.

How to trace MSMQ?

I have an agent and a server in different domains. The server acts as an MSMQ server and the agent acts as an MSMQ client. I am using the mqsender utility, which is part of the MSMQ tools.
My problem is that a message is not delivered when using the HTTP:// format string (the MSMQ is installed with HTTP support). Using the OS: format string works fine.
When using HTTP the messages are immediately moved to the Dead Letter queue and the Class is set to Unknown, so I do not know the reasons for this behaviour.
So, this works:
mqsender.exe /c:10 /j:dead /f:Direct=OS:il-mark-w2k3\private$\test
And this does not:
mqsender.exe /c:10 /j:dead /f:Direct=http://il-mark-w2k3/msmq/private$/test
I checked that MSMQ virtual directory exists. How can I trace the MSMQ operation to try and understand what is going on?
Thanks.
EDIT
All the commands work as expected when ran locally on the server.
Navigating to http://il-mark-w2k3/msmq/private$/test in the browser on the agent (and the server) results in 501 - Header values specify a method that is not implemented. The same error is received when navigating to http://il-mark-w2k3/msmq. I suppose that is OK, after all it is not 404 - Not Found, right?
EDIT2
I have succeeded to resolve the issue. IIS lacked Anonymous Authentication, it became obvious from observing its log - 401.2 HTTP error was there. All worked well after it was enabled. The mistery remains why did MSMQ display Class Unknown on the dead messages? On other machine the same setup produces Error : 401, which makes much more sense.
The logging for MSMQ is internal so you won't easily be able to see exactly why the message didn't get delivered without raising a support case with Microsoft.
I have a few blog posts on solving various MSMQ/HTTP issues.
The 17 entitled "MSMQ messages using HTTP just won't get delivered" may help.
Also make sure you check the IIS logs for information.
Cheers
John Breakwell