We have a set of micro services which obtain their configuration from a configuration server that we have created. Our configuration server uses mongo as the backing store for configuration files and we run it in a redundant configuration so that if any of the configuration servers fail then we can fail over to another.
Now the configuration for each service is that it uses discovery to get the location of the configuration server. We can see that each service obtains a reference ants the proper configuration data for the server and that all services start correctly with this data.
Now one thing that we noticed is that the services regularly request their configurations from the configuration server every 3 mins. This is being done at the configuration client level and not in our code base.
This get request from every service at that rate is causing quite a bit of load on the configuration service. Is this expected and if so how can be lower the rate of these pings?
One additional issue is that if we make a health request to any service, this triggers a request to get the services configuration from the remote configuration service.
The combination of both of these elements is driving our configuration server into the ground since they are 5 requests/sec on the server.
Related
I am trying setting up an websphere application server cluster and deploy an web application on it. I have successfully access the application with a web server. but i have some question, if i would access the application bypassing the web server by using 9080 to the single server, i get error when sign in servlet respond.redirect to jsp page. the session was lost, are we not suppose to access the application in cluster environment with out the web server ?
As long as you target an individual server in the cluster, you should not lose your session or suffer any other ill effects for not using/having a proxy server between you and the cluster. You should look closely at the Cookies issued through that flow and make sure they don't have a bad domain or path and that you don't get redirected back through the webserver or otherwise to another JVM.
Also verify there is no DNS-based load balancing. If you don't have session persistence/distribution/replication it won't work.
Background
We have almost a hundred Eureka clients (spring cloud apps) being registered to an Eureka discovery server. One client can depend as many as more than 10 other clients. Traditionally, we have to run all the dependencies (along with discovery and config server) on local dev environment when developing against one client. Sometimes, to avoid bring up so many projects, I just connect the single client I am working on to our integration test environment which has all clients running. This way, it is very convenient for myself, but having a critical issue: my local client can be discovered by other clients, meaning someone else's testing might be effected by my instance. I know there are ways to mock or proxy the client dependencies. But they are not always good for every scenario, also the setup takes additional time.
Question
Is there a practically quick and easy way to register a client without being discovered by other clients. Besides, I can not issuePUT /eureka/v2/apps/appID/instanceID/status?value=OUT_OF_SERVICEor other discovery server side command. I can only configure my local project.
The easiest way to achieve that is adding the below property only in your local profile.
eureka:
client:
registerWithEureka: false
If so, your application still can look up other instances from Eureka, but your application will not register itself into Eureka.
I'm trying to authenticate on my web application deployed on a jboss working in cluster mode with 2 nodes.
After a succesful authentication I get redirected to an admin page where a Filter checks if I am logged in.
On standalone mode it works just fine but when I deployed into production, which uses cluster mode, the filter rejects my request because it can't access the session parameters I have established on authentication.
Using the developer tools I see there are 3 JSESSIONID cookies set: one for /, one for /myapplication path and another one called JSESSIONID-34234 also for /myapplication path (I've cleared all them before starting the process).
Browsing the jboss docs I can't see no explanation for this although it seems the source of my problem.
How can I get to work authentication (I am using spring security http form based authentication) in my JBoss cluster?
Solved by enabling sticky session by adding the following to the virtualhost configuration file:
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/myapplication" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://jboss6-hc-001-8109>
BalancerMember ajp://jboss2.imatiasl.lan:8109 route=jboss2-hc-001-server-02
BalancerMember ajp://jboss3.imatiasl.lan:8109 route=jboss3-hc-001-server-02
ProxySet lbmethod=byrequests stickysession=ROUTEID
</Proxy>
Web session clustering should work if:
You enabled <distributed/> in web.xml.
Your app's server group is using ha or full-ha profile
If you want your clustered app perform better, consider implementing a good load-balancing policy. For most webapps load-balancing with sticky sessions is OK.
In some webapps, it is enough not to ask for re-authentication in case of failover or session is very easy to rebuild if authentication info is available. In such cases you even don't need web session clustering. Clustered SSO is enough, the caveat is you'll have to use container level security for authentication (most probably supported by spring-security). This way only authentication info is replicated, so you'll have to design session data management to be resilient to situations, when session suddenly becomes empty.
In our application, We have 6 instance of application server.
When we test it manually, we will land on some node randomly.
When i run our JMeter test (with 20 users), most of the load(70%) goes to 1 specific node out of 6 nodes.
30% goes to another specific node from the rest 5.
I see the information via JSESSIONID set in the request cookies.
Issue:
I was thinking the load will be balanced on all the 6 nodes equally (more or less).
Is this my application issue? Or Does it have to do anything with JMeter?
Note:
Our app server is JBoss. It is a cookie injection based load balancing.
I do not send the request to any specific app server via cookie. The cookie is set automatically.
First of all make sure that HTTP Cookie Manager is added to your test plan, it should be enough to automatically resolve cookie-based load balancing problem.
If you have > 1 load balancer instance or the load balancer resolves in > 1 IP address you may have to add a DNS Cache Manager as well. See The DNS Cache Manager: The Right Way To Test Load Balanced Apps guide for better explanation and configuration details.
Beside what Dmitri suggested you should check the way your LB is configured.
In many cases, the LB distributes the requests based on source IP, so all requests coming form the Jmeter server go to the same target.
This happens especially when the LB does not terminate the https traffic.
I'm using Windows Azure Service Bus relay to expose a WCF service that lives on premises using basicHttpRelayBinding.
For every request, the web service will just get some data out of a SQL Database using entity framework, then convert the results to an IEnumerable and finally send the response to the client.
Everything works just fine and I'm able to consume this web service from our web and worker roles hosted in the cloud, but every once in a while I'm getting the 504 Gateway timeout error while waiting for a response from the service, Typically this only happens with queries that take more than 30 - 40 seconds to execute.
Been doing some research and I've find out that this might be due to slow communication between a server acting as a gateway/proxy and some server upstream in the chain, but I'm not sure if there's any way to increase the timeout value so we don't get the 504 error.
Any help would be really appreciated.