ServiceProxy throws ProtocolException, communication is not restored on retrying - azure-service-fabric

We are seeing ProtocolExceptions while communicating with a service running in the cluster. The message and InnerException message:
System.ServiceModel.ProtocolException: You have tried to create a channel to a service that does not support .Net Framing.
---> System.IO.InvalidDataException: Expected record type 'PreambleAck', found '145'.
This service is running on a local dev cluster, and the exception is thrown after communicating successfully with the service.
The code that we use for communicating is:
var eventHandlerServiceClient = ServiceProxy.Create<IEventHandlerService>(eventHandlerTypeName, new Uri(ServiceFabricSettings.EventHandlerServiceName));
return await eventHandlerServiceClient.GetQueueLength();
We have retry logic (with increasing delay's between the attempts). But this call never succeeds. So it looks like the service is in a fault state and cannot recover from it.
Update
We are also seeing the following errors in the logs:
connection 0x1B6F9EB0 localhost:64002-[::1]:50376 target 0x1B64F3C0: invalid frame: length=0x1000100,type=514,header=28278,check=0x742E7465
Update 14-12-2015
If this ProtocolException is thrown, retries don't help. Even after hours of waiting, it still fails.
We log the endpoint address with
var spr = ServicePartitionResolver.GetDefault();
var x = await spr.ResolveAsync(new Uri(ServiceFabricSettings.EventHandlerServiceName),
eventHandlerTypeName,
new CancellationToken());
var endpointAddress = x.GetEndpoint().Address;
The resolved endpoint looks like
{"Endpoints":{"":"net.tcp:\/\/localhost:57999\/d6782e21-87c0-40d1-a505-ec6f64d586db\/a00e6931-aee6-4c6d-868a-f8003864a216-130945476153695343"}}
This endpoint is the same as reported by the Service Fabric Explorer.
From our logs seen, it seems that this service is working (it is reachable via another API method), but this specific call never succeeds.

This typically indicate mismatched communication stack on the service and client side. Once the service is up and running, check the endpoint of the service replica via Service Fabric Explorer. If that seems fine, check that the client is connecting to the right service. Resolve the partition using the ServicePartitionResolver (https://msdn.microsoft.com/en-us/library/azure/microsoft.servicefabric.services.servicepartitionresolver.aspx), passing the same arguments that you pass to ServiceProxy.

I'm seeing the same sort of errors. Just looking at my code, I'm caching an actorproxy. I'm going to change that and remove the caching in case the cache is referencing an old instance of the service.
That appears to have fixed my issues. I'm guessing that the proxy caches the reference once it has been used and if the service changes, that reference is out of date.

Related

Keycloak Cached clientScope not found

Getting repeatedly this error in Keycloak logs.
Attached below are the logs for reference:
The said client scope is not found if I try to search the same under Keycloak admin console too.
2022-09-22 04:04:12,718 ERROR [org.key.ser.err.KeycloakErrorHandler] (executor-thread-610) Uncaught server error: java.lang.IllegalStateException: Cached clientScope not found: 1e84ef04-9ef9-44fe-b1bd-f45e6d4 at org.keycloak.models.cache.infinispan.RealmAdapter.lambda$getClientScopesStream$3(RealmAdapter.java:1495) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) at
Steps to recreate:
Create client scope and attached to a client
Remove client scope attached with a client
Delete client scope
On the occurrence of this issue, I see well-known and many other endpoints to start giving 500 as they seem to be fetching cached scoped and doing certain validation
Any configuration miss that needs to be considered?

Postman : socket hang up

I just started using Postman. I had this error "Error: socket hang up" when I was executing a collection runner. I've read a few post regarding socket hang up and it mention about sending a request and there's no response from the server side and probably timeout. How do I extend the length of time of the request in Postman Collection Runner?
For me it was because my application was switched to https and my postman requests still had http in them. Changing postman to https fixed it.
Socket hang up, error is port related error. I am sharing my experience. When you use same port for connecting database, which port is already in use for other service, then "Socket Hang up" error comes out.
eg:- port 6455 is dedicated port for some other service or connection. You cannot use same port (6455) for making a database connection on same server.
Sometimes, this error rises when a client waits for a response for a very long time. This can be resolved using the 202 (Accepted) Http code. This basically means that you will tell the server to start the job you want it to do, and then, every some-time-period check if it has finished the job.
If you are the one who wrote the server, this is relatively easy to implement. If not, check the documentation of the server you're using.
Postman was giving "Could not get response" "Error: socket hang up".
I solved this problem by adding the Content-Length http header to my request
Are you using nodemon, or some other file-watcher? In my case, I was generating some local files, uploading them, then sending the URL back to my user. Unfortunately nodemon would see the "changes" to the project, and trigger a restart before a response was sent. I ignored the build directories from my file-watcher and solved this issue.
Here is the Nodemon readme on ignoring files: https://github.com/remy/nodemon#ignoring-files
I have just faced the same problem and I fixed it by close my VPN. So I guess that's a network agent problem. You can check if you have some network proxy is on.
this happaned when client wait for response for long time
try to sync your API requests from postman
then make login post and your are done
I defined Authenticate method to generate a token and mentioned its return type as nullable string as:
public string? Authenticate(string username, string password)
{
if(!users.Any(u => u.Key==username && u.Value == password))
{
return null;
}
var tokenHandler = new JwtSecurityTokenHandler();
var tokenKey = Encoding.ASCII.GetBytes(key);
var tokenDescriptor = new SecurityTokenDescriptor()
{
Subject = new ClaimsIdentity(new Claim[]
{
new Claim(ClaimTypes.Name, username)
}),
Expires = DateTime.UtcNow.AddHours(1),
SigningCredentials = new SigningCredentials(new
SymmetricSecurityKey(tokenKey),
SecurityAlgorithms.HmacSha256Signature)
};
var token = tokenHandler.CreateToken(tokenDescriptor);
return tokenHandler.WriteToken(token);
}
Changing nullable string to simply string fixed "Socket Hang Up" issue for me!
If Postman doesn't get response within a specified time it will throw the error "socket hang up".
I was doing something like below to achieve 60 minutes of delay between each scenario in a collection:
get https://postman-echo.com/delay/10
pre request script :-
setTimeout(function(){}, [50000]);
I reduced time duration to 30 seconds:
setTimeout(function(){}, [20000]);
After that I stopped getting this error.
I solved this problem with disconnection my vpn. you should check if there is vpn connected.
What helped for me was replacing 'localhost' in the url to http://127.0.0.1 or whatever other address your local machine has assigned localhost to.
Socket hang up error could be due to the wrong URL of the API you are trying to access in the postman. please check the URL once carefully.
It's possible there are 2 things, happening at the same time.
The url contains a port which is not commonly used AND
you are using a VPN or proxy that does not support that port.
I had this problem. My server port was 45860 and I was using pSiphon anti-filter VPN. In that condition my Postman reported "connection hang-up" only when server's reply was an error with status codes bigger than 0. (It was fine when some text was returning from server with no error code.)
When I changed my web service port to 8080 on my server, WOW, it worked! even though pSiphon VPN was connected.
Following on Abhay's answer: double check the scheme. A server that is secured may disconnect if you call an https endpoint with http.
This happened to me while debugging an ASP.NET Core API running on localhost using the local cert. Took me a while to figure out since it was inside a Postman environment and also it was a Monday.
In my case, adding in the header the "Content-length" parameter did the job.
My environment is
Mac:
[Terminal command: sw_vers]
ProductName: macOS
ProductVersion: 12.0.1. (Monterey)
BuildVersion: 21A559
mysql:
[Terminal command: mysql --version]
Ver 8.0.27 for macos11.6 on x86_64 (Homebrew)
Apache:
[Terminal command: httpd -v]
Server version: Apache/2.4.48 (Unix)
Server built: Oct 1 2021 20:08:18.
*Laravel
[Terminal command: php artisan --version]
Laravel Framework 8.76.2
Postman
Version 9.1.5 (9.1.5)
socket hang up error can also occur due to backend API handling logic.
For example - I was trying to create an Nginx config file and restart the service by using the incoming API request body. This resulted in temporary disconnection of the Nginx service while handling the API request and resulted in socket hang up.
If you have tried all the steps mentioned in other comments, and still face the issue. I suggest you check the API handler code thoroughly.
I handled the above-mentioned example by calling the Nginx reset method with delay and a separate API to check the status of the prev reset request.
For me it was giving Socket Hung Up error only while running Collection Runner not with single request.
Adding a small delay (100-300ms) in the collection Runner solved issue for me.
In my case, I had to provide --ssl-client-key and --ssl-client-cert files to overcome these errors.
Great error, it is so general that for everyone something different helps.
In my case I was not able to fix it and what is really funny is fact that I am expecting to get multipart file on one endpoint. When I prepare request in postman I get "Error: socket hang up". If I change for other endpoint(even not existing) is exactly that same error. But when I call any endpoint without body that request works and after that all subsequent attempts works perfectly.
In my case this is purely postman issue. Any request using curl is never giving that error.
For me the issue was related to the mismatch of the http versions on the client and server.
Client was assuming http v2 while server (spring boot/ tomcat) in the case was http v1
When on the server I configured server to v2, the issue got resolved in a go.
In spring boot you can configure the http v2 as below:-
server.http2.enabled=true
Note - Also the scenario was related to using client auth mechanism (i.e. MTLS)
Without client auth/ MTLS it worked without issues but for client auth the version setting in spring boot was the important rescue point
"socket hang up" is proxy related issue. when we run same collection with the help of newman on jenkins then all test are passed.
change the proxy setting
https://docs.cloudfoundry.org/cf-cli/http-proxy.html
I had the same issue: "Error: socket hang up" when sending a request to store a file and backend logs mentioned a timeout as you described. In my case I was using mongoDB and the real problem was my collection’s array capacity was full. When I cleared the documents in that collection the error was dismissed. Hope this will help someone who faces a similar scenario.
"Socket Hung Up" can be on-premise issue some time's, because, of bottle neck in %temp% folder, try to free up the "temp" folder and give a try
I fixed this issue by disabling Postman token header. Screenshot:
I face the same issue in when calling a SOAP API with POSTMAN
by adding the following data in the header my issue was fixed
Key:Content-Length
Value:<calculated when request is sent>
In my case, I was incorrectly using a port reserved for https version of my api.
For example, I was supposed to use https://localhost:6211, but I was using http://localhost:6211.
It is port related error. I was trying to hit the API with an invalid port.
if it helps to anybody... In my case, i just forgot to use json parser (const jsonParser = express.json();) to have access to json type of objects sending to the server from the client. Be careful, don't waste your time =)
This happened to me while I was learning ASP.NET Web API.
In my case it was because the SSL certificate verification.
I was using VS Code so I oversee about SSL certificate verification and it came with https protocol.
I solved this with testing my endpoints with http protocol.
Another approach can be just disabling the SSL certificate Verification on Postman Settings.
This error was coming for me since the request url is not correct --> here you can see my url does not contains : after http
The url I was using was : http//locahost:9090/someApi
Solution
adding a colon new url is http://localhost:9090/someApi
the socket error was not coming
This is just my case may be your case is totally different as mentioned in the other answers :)

Clearing a faulty service in WSo2

I have recently built a proxy service in WSo2 ESB that was not fully implemented. When I saved it generated a fault service message and a link was provided to a faulty service group at the top of the ESB console. Since then, I have corrected the service and it has been transferring files as I have intended. However the service does not show in the initial list of services and I have to click on the link 7 deployed service group(s) to now view it in the proxy list.
Here is what the link looks like:
6 active services. 7 deployed service group(s). 1 faulty service(s).
When I click on the 1 faulty service(s) I see the following but cannot delete it in an attempt to clear it.
*Faulty Service Actions
RenZipExtractProxy proxy
Unable to configure the service RenZipExtractProxy for the VFS transport: Service doesn't have configuration information for transport vfs. This service is being marked as faulty and will not be available over the VFS transport.*
How do I clear this faulty service issue? My updated service works fine but I continue to get the faulty service situation as stated above.
You can manually delete deployment artifacts in the directory <ESB>/repository/deployment/server/synapse-configs/default/proxy-services
I have noticed that when we configure WSO2 esb with VFS protocol and do not provide any parameter then it is going under Faulty service. I resolved this problem after mentioning VFS related parameters.
Please refer https://docs.wso2.com/display/ESB481/VFS+Transport
Note: If we try to delete faulty proxy service from console then also it will not deleted. Please try to delete it from file system.
<ESB_HOME>/repository/deployment/server/synapse-configs/default/proxy-services

MessageQueue.Exists(QueueName) returns false but it exists

The problem I'm having is with this code:
if (!MessageQueue.Exists(QueueName))
{
MessageQueue.Create(QueueName, true);
}
It will check if a queue exists; if it doesn't I want it to create the queue. This code has been working and hasn't changed for a few months. Today I started receiving this error:
[MessageQueueException (0x80004005): A queue with the same path name
already exists.] System.Messaging.MessageQueue.Create(String path,
Boolean transactional) +239478
The queues are local and if I delete the specific queue it will work once. After the queue is created it starts to fail again with the same error message.
It looks like the issue may be because of the Network Load Balancing (NLB) configuration. I was unaware of a change that recently put the machine in a NLB environment. The configuration we are using is an unsupported one.
More information is in How Message Queuing can function over Network Load Balancing (NLB).

OpenStack API CreateInstance

After I create an instance on OpenStack with the REST API Using the Nova endpoint the operation succeeds with the server id reference. If I immediately try to get any of the details of the network interfaces (using the /servers/server-id/ips) I get nothing, not even an error message that the resource is busy. If I put an arbitrary 30 second wait, I'll get the details. Is there any API call that can use on OpenStack to get a "is ready" state after a creation?
If there is no operation to determine that the server is ready, then is there a recommended wait time in the documentation?
You should wait for the instance to be in the "active" state. That's when the instance has been fully built, including network. You're probably hitting the instance while its still in the "building" state.