I'am working on a microservice architecture based on Docker, registrator, consul and HAProxy.
I'am also using Consul-template to dynamically generate the HAProxy config file. Everything works fine : When I add multiple instances of the same microservice, the HAProxy configuration is updated immediately and requests are dispatched correctly using a round robin strategy.
My problem occurs when I remove some instances (scale down). If a container is shut down while a request is running I have an error.
I'am new to HAProxy so is there a way to configure HAProxy to tell it to retry a failing request to another endpoint if a container disappears?
Precision : I'am using a layer7 routing mode (mode http) for my frontends and backends. Here is a little sample of my consul-template file :
backend hello-backend
balance roundrobin
mode http
{{range service "HelloWorld" }}server {{.Node}} {{.Address}}:{{.Port}} check
{{end}}
# Path stripping
reqrep ^([^\ ]*)\ /hello/(.*) \1\ /\2
frontend http
bind *:8080
mode http
acl url_hello path_beg /hello
use_backend hello-backend if url_hello
Thank you for your help.
It isn't possible for HAProxy to resend a request that has already been sent to a backend.
Here's a forum post from Willy, the creator.
redispatch only happens when the request is still in haproxy. Once it has been sent, it is cannot be performed. It must not be performed either for non idempotent requests, because there is no way to know whether some processing has begun on the server before it died and returned an RST.
http://haproxy.formilux.narkive.com/nGKXq6WU/problems-with-haproxy-down-servers-and-503-errors
The post is quite old but it's still applicable based on more recent discussions. If a request is larger than tune.bufsize (default is around 16KB iirc) then HAProxy hasn't even retained the entire request in memory at the point an error occurs.
Both fortunately (for the craft) and unfortunately (for purposes of real-world utility), Willy has always insisted on correct behavior by HAProxy, and he is indeed correct that it is inappropriate to retry non-idempotent requests once they have been sent to a back-end server, because there are certainly cases where this would result in duplicate processing.
For GET requests which, by definition, should be idempotent (a GET request must be repeatable without consequence, otherwise it should not have been designed to use GET -- it should have been POST or another verb) there's a viable argument that resending to a different back-end would be a legitimate course of action, but this also is not currently supported.
Varnish, by contrast, does support a do-over, which I have used (behind HAProxy) with success on GET requests where I have on-line and near-line storage for the same object namespace. Old, "unpopular" files are migrated to near-line (slower, cheaper) storage, but all requests are sent to on-line storage, with the retry destination of near-line if on-line returns a 404. But, I've never tried this with requests other than GET.
Ideally, your solution would be for your back-ends to be declared unhealthy, perhaps by deliberately failing their HTTP health checks for a draining time before shutting down. One fairly simple approach is for the health check to require the presence of a static file, which gets deleted from the back-end before shutdown. Or, you can request HAProxy consider the backend to be in maintenance mode through the stats/admin UI or socket, preventing more requests from being initiated while allowing running requests to drain.
Related
The manual says this about the HTTP URL value of an http listener:
"Displays the generated HTTP URL for the HTTP Listener. This is not an actual
configurable setting, but is instead displayed for copy/paste convenience. Note
that the host in the URL will be the same as the host you used to connect to
the Administrator. The actual host that connecting clients use may be different
due to differing networking environments."
When I have used the feature in the past its value has always begun "http://localhost:" which would be great except this time it is auto-generating " http://'domainName':${Incoming_Pathology_Source_Port}/${Incoming_Pathology_Source_BaseContextPath}/"
For the first time, we are deploying Mirth inside a Kubernetes cluster, 'a different working environment'. (nginx accepts https and we want it pass the messages on as http to Mirth).
Is there any way I can take control of the URL or must I change the configuration of the cluster in some way.
All help/suggestions welcome.
When watching kubernetes resources for changes, what exactly is happening under the hood? Does the http suddenly change to a wss connection?
To solve a problem of too many requests to the kube-apiserver I am rewriting some code to what I think is more of an operator pattern.
In our multi-tenant microservice architecture all services use the same library to look up connection details to tenant-specific DBs. The connection details are saved in secrets within the same namespace as the application. Every tenant DB has its own secret.
So on every call all secrets with the correct label are read and parsed for the necessary DB connection details. We have around 400 services/pods...
My idea: instead of reading all secrets on every call, create a cache and update the cache everytime a relevant secret was changed via a watcher.
My concerns: am I just replacing the http requests with equally expensive websockets? As I understand I will now have an open websocket connection for every service/pod, which still is 400 open connections.
Would it be better to have a proxy service to watch the secrets (kube-apiserver requests) and then all services query that service for connection details (intranet requests, kube-apiserver unreleated)?
From the sources:
// ServeHTTP serves a series of encoded events via HTTP with Transfer-Encoding: chunked
// or over a websocket connection.
It pretty much depends on the client which protocol is used (either chunked http or ws), both of them having their cost, which you'll have to compare to your current request frequency.
You may be better of with a proxy cache that either watches or polls in regular intervals, but that depends a lot on your application.
I'd like to have health checks to an haproxy instance fail solely based on whether haproxy is running. In other words I don't want the health check to be proxied to a backend server.
I see there is a way to do this by returning a static file. This would work for me, but I was wondering: Is there a way to return an empty response with just a status code without having to return a file and deal with the false 503? The solution linked seems hacky and I worry that behavior will not be allowed in some later version.
monitor-uri <uri>
Intercept a URI used by external components' monitor requests
May be used in sections :
When an HTTP request referencing will be received on a frontend,
HAProxy will not forward it nor log it, but instead will return either
"HTTP/1.0 200 OK" or "HTTP/1.0 503 Service unavailable", depending on
failure conditions defined with "monitor fail". This is normally
enough for any front-end HTTP probe to detect that the service is UP
and running without forwarding the request to a backend server. Note
that the HTTP method, the version and all headers are ignored, but the
request must at least be valid at the HTTP level. This keyword may
only be used with an HTTP-mode frontend.
Example :
# Use /haproxy_test to report haproxy's status
frontend www
mode http
monitor-uri /haproxy_test
I have a system with multiple machines part of mongo sharded cluster. There is REST api to access the database and has GET POST methods to do so.
There is ha-proxy to load balance the HTTP requests across these machines.
In order to boost performance i am planning to tweak the behavior in such a way that all the POST REST requests (which are comparatively less in number) to PRIMARY instance and all the GET request getting load balanced through ha-proxy.
I tried various configuration schemes but i am not able to understand how to expose the POST logic to ha-proxy.
Current setup is as follows
CLIENT Machine - a simulator to generate REST requests
Multiple Server machines part of mongo sharded cluster
A machine running ha-proxy
this is my ha-proxy configuration
frontend main *:5000
bind 10.60.100.100:8080
default_backend serverpool
backend serverpool
balance roundrobin
mode tcp
server sv1 7.7.7.1:8080 check
server sv2 7.7.7.2:8080 check
In our application, We have 6 instance of application server.
When we test it manually, we will land on some node randomly.
When i run our JMeter test (with 20 users), most of the load(70%) goes to 1 specific node out of 6 nodes.
30% goes to another specific node from the rest 5.
I see the information via JSESSIONID set in the request cookies.
Issue:
I was thinking the load will be balanced on all the 6 nodes equally (more or less).
Is this my application issue? Or Does it have to do anything with JMeter?
Note:
Our app server is JBoss. It is a cookie injection based load balancing.
I do not send the request to any specific app server via cookie. The cookie is set automatically.
First of all make sure that HTTP Cookie Manager is added to your test plan, it should be enough to automatically resolve cookie-based load balancing problem.
If you have > 1 load balancer instance or the load balancer resolves in > 1 IP address you may have to add a DNS Cache Manager as well. See The DNS Cache Manager: The Right Way To Test Load Balanced Apps guide for better explanation and configuration details.
Beside what Dmitri suggested you should check the way your LB is configured.
In many cases, the LB distributes the requests based on source IP, so all requests coming form the Jmeter server go to the same target.
This happens especially when the LB does not terminate the https traffic.