I am wondering if there is a way to configure Eureka server in Spring Cloud to attempt to restart the clients which are crashed. I couldn't find anything in the documentation, but I think at some point Eureka was capable of restarting the failed clients
No. Eureka is not involved in starting, stopping or scaling services.
Related
I have an app hosted on GKE which, among many tasks, serve's a zip file to clients. These zip files are constructed on the fly through many individual files on google cloud storage.
The issue that I'm facing is that when these zip's get particularly large, the connection fails randomly part way through (anywhere between 1.4GB to 2.5GB). There doesn't seem to be any pattern with timing either - it could happen between 2-8 minutes.
AFAIK, the connection is disconnecting somewhere between the load balancer and my app. Is GKE ingress (load balancer) known to close long/large connections?
GKE setup:
HTTP(S) load balancer ingress
NodePort backend service
Deployment (my app)
More details/debugging steps:
I can't reproduce it locally (without kubernetes).
The load balancer logs statusDetails: "backend_connection_closed_after_partial_response_sent" while the response has a 200 status code. A google of this gave nothing helpful.
Directly accessing the pod and downloading using k8s port-forward worked successfully
My app logs that the request was cancelled (by the requester)
I can verify none of the files are corrupt (can download all directly from storage)
I believe your "backend_connection_closed_after_partial_response_sent" issue is caused by websocket connection being killed by the back-end prematurily. You can see the documentation on websocket proxying in nginx - it explains the nature of this process. In short - by default WebSocket connection is killed after 10 minutes.
Why it works when you download the file directly from the pod ? Because you're bypassing the load-balancer and the websocket connection is kept alive properly. When you proxy websocket then things start to happen because WebSocket relies on hop-by-hop headers which are not proxied.
Similar case was discussed here. It was resolved by sending ping frames from the back-end to the client.
In my opinion your best shot is to do the same. I've found many cases with similar issues when websocket was proxied and most of them suggest to use pings because it will reset the connection timer and will keep it alive.
Here's more about pinging the client using WebSocket and timeouts
I work for Google and this is as far as I can help you - if this doesn't resolve your issue you have to contact GCP support.
We have an application that's receiving calls from other services that use Eureka to discover the different IP addresses of the different application instances/replicas.
When deploying a new version of this app, our deployment system (kubernetes in our case) sends a SIGTERM to one of the instances of the application to shut it down.
But the Eureka client in the services sending requests to the application, keeps a local cache of Eureka's information. Meaning that these applications won't realize that the instance of the app has been shutt down, and they will continue to send requests to an instance that is no longer working.
Is there a way to make a Spring Cloud application to wait for some seconds before shutting down to make sure that all clients have the updated Eureka information (where this app won't be listed anymore)?
If you're using Kubernetes then you could map a Service to each of the apps/services that register with eureka and tell the apps to register using the service name instead of an IP. Then you can manage the blue/green deploy with Kubernetes (provided you've got probes set up). Eureka will then just know about the Service names and whether something is registered for them and Kubernetes will be managing availability during the upgrade. It's a bit of a hybrid model.
If you are removing an app or changing the name rather than upgrading then I think you'll need to set a lease time for the eureka registration data. Eureka never unregisters a service
As part of the move from Eureka to consul (Spring Cloud), we are looking at replicating certain behaviours.
One of them is that if a service disconnected to Eureka after a significant will reconnect when it can see Eureka again.
I see that a service will retry to consul and eventually stop. Is there a way to replicate the same behaviour on a service to consul past the max retries/backoff?
Just trying to support expected behaviours
It's a known issue https://github.com/spring-cloud/spring-cloud-consul/issues/197. That functionality you describe is built into the eureka thick client and something similar would need to be built for spring-cloud-consul.
Currently our application uses a hardware load balancer for SSL termination. We are beginning to implement a new architecture that breaks the one big application into a set of smaller ones, likely using the Netflix & Spring Cloud tools.
As we look at Zuul one of the questions from the operations team is where are people usually terminating the SSL connections when having lots of connections (and Zuul services)? We did in the load balancer years ago due to the CPU costs of terminating the SSL on the single application server, but if we deploy a set of services on multiple machines does that remove that concern?
So where are people terminating their SSL when using Zuul?
Thanks,
Chris
We have been struggling with this same question. I hate to put yet another proxy in front of Zuul. If it really is an edge service and can handle global load balancing and other concerns I would think that Zuul is the likely place to handle SSL.
For now we are going to use an F5 because we have them available. You could also use a HAProxy software load balancer, NGinX or Apache.
If you are using Zuul as proxy, what's wrong terminating tls/ssl at zuul? (Assuming all requests goes thru zuul).
I want to be able to stop discovery on a running service and potentially keep it from joining with eureka for an misbehaving service. I can't see a specific property that will allow me to do this.
Basically a service is in the mix and causing issues and I want to have either a) the service stop registering to eureka or b) cancel a registration on eureka for a service.
Any ideas?
If it is not possible let me know and i will spin my own version of the discovery mechanism.