I have setup a service that uses RegisterServiceNotificationFilterAsync to get notified of service change events. It works as intended. When a service goes down, this event gets called.
But it happens AFTER the service has gone offline. Which means that several requests could have failed against that now offline service before I get it pulled out of my loadbalancer pool.
Sometimes Service Fabric can only react to a service going offline. For example, if someone pulls the plug on a server node, Service Fabric clearly can't tell me in advance that the service is going offline.
But many times, a threshold is reached and it is Service Fabric itself that kills the service (and starts a new one).
Is there a way to know BEFORE Service Fabric kills a service? (So I have time to update my loadblancer.)
(In case it matters, I am running Service Fabric on premises.)
You can only be notified on shutdown inside the service. RegisterServiceNotificationFilterAsync is based on endpoint changes from the naming service.
If it's a reliable service, you get events for these scenarios: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-lifecycle
For guest executeables the signal Service Fabric send is Ctrl+C. For containers it's "docker stop".
Related
Recently the our Azure service fabric health went bad after a deployment. The deployment was successful but service fabric health went bad due to some code issue and it was not rolling back. Only on looking into the service fabric explorer did we know that the cluster went bad
Is there a way to get an email alert when the service fabric health goes bad.
Scenarios where service fabric failed
Whole cluster so what happened was 1 service went bad(showed in red) and was consuming a lot of memory and that in turn caused other services to go bad. after which the whole cluster I had to log into the scaleset to see which services was taking most of the memory.
In another case we added another reliable collection to existing reliable collection to statefull service. This caused failure.
in each of the cases i need to look at the servifabric explorer and then go to each scale set to see the actual error message.
We have an application that's receiving calls from other services that use Eureka to discover the different IP addresses of the different application instances/replicas.
When deploying a new version of this app, our deployment system (kubernetes in our case) sends a SIGTERM to one of the instances of the application to shut it down.
But the Eureka client in the services sending requests to the application, keeps a local cache of Eureka's information. Meaning that these applications won't realize that the instance of the app has been shutt down, and they will continue to send requests to an instance that is no longer working.
Is there a way to make a Spring Cloud application to wait for some seconds before shutting down to make sure that all clients have the updated Eureka information (where this app won't be listed anymore)?
If you're using Kubernetes then you could map a Service to each of the apps/services that register with eureka and tell the apps to register using the service name instead of an IP. Then you can manage the blue/green deploy with Kubernetes (provided you've got probes set up). Eureka will then just know about the Service names and whether something is registered for them and Kubernetes will be managing availability during the upgrade. It's a bit of a hybrid model.
If you are removing an app or changing the name rather than upgrading then I think you'll need to set a lease time for the eureka registration data. Eureka never unregisters a service
I'm in the middle of creating an K8s app that doesn't expose any HTTP endpoints, is just a background app that pulls messages from a message bus and takes some action based on the incoming message. No other apps will interact directly with this background app, only thru posting messages into the message bus.
Scaling is a requirement and most likely will always need to run more than one replica. What is the recommended Service type in Kubernetes to handle this type of workload ?
No service required... just create a Deployment, which will result in a ReplicaSet, which will keep n replicas of your app running.
When I make an RPC (service remoting) call to a service that is deployed on multiple nodes from another service in the same application, it appears to be going to all nodes at once. I only want it to go to one each time the call is made.
Does Service Fabric have a way to do that? How can I leverage the built-in load balancing to control where the call goes to?
This deployed on a local cluster
If your service is stateless and uses Singleton partitioning, calling an operation using the ServiceProxy will invoke the operation on one random service instance. Using SF remoting, you can't target a specific instance.
If your service is stateful, calling an operation using the ServiceProxy (created with a specific ServicePartitionKey) will invoke the operation on one of the replicas of your service, using the primary replica by default.
I have a scenario where one of our services exposes WCF hosts that receive callbacks from an external service.
These hosts are dynamically created and there may be hundreds of them. I need to ensure that they are all up and running on the node before the node starts receiving requests so they don't receive failures, this is critical.
Is there a way to ensure that the service doesn't receive requests until I say it's ready? In cloud services I could do this by containing all this code within the OnStart method.
My initial thought is that I might be able to bootstrap this before I open the communication listener - in the hope that the fabric manager only sends requests once this has been done, but I can't find any information on how this lifetime is handled.
There's no "fabric manager" that controls network traffic between your services within the cluster. If your service is up, clients or other services inside the cluster can choose to try to connect to it if they know its address. With that in mind, there are two things you have control over here:
The first is whether or not your service's endpoint is discoverable by other services or clients. This is the point at which your service endpoint is registered with Service Fabric's Naming Service, which occurs when your ICommunicationListener.OpenAsync method returns. At that point, the service endpoint is registered and others can discover it and attempt to connect to it. Of course you don't have to use the Naming Service or the ICommunicationListener pattern if you don't want to; your service can open up an endpoint whenever it feels like it, but if you don't register it with the Naming Service, you'll have to come up with your own service discovery mechanism.
The second is whether or not the node on which your service is running is receiving traffic from the Azure Load Balancer (or any load balancer if you're not hosting in Azure). This has less to do with Service Fabric and more to do with the load balancer itself. In Azure, you can use a load balancer probe to determine whether or not traffic should be sent to nodes.
EDIT:
I added some info about the Azure Load Balancer to our documentation, hope this helps: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-connect-and-communicate-with-services/