We are using Hazelcast 4 to implement distributed locking across two pods on kuberentes.
We have developed distributed application, two pods of micro service has been created. Both instances are getting auto discovered and forming members.
We are trying to use IMap.lock(key) method to achieve distributed locking across two pods however both pods are acquiring lock at same time, thereby executing the business logic at the concurrently. Also hazelcast management center shows zero locks for the created Imap.
Can you please help on how to achieve synchronization of imap lock(key) so that single pod get the lock for given key at given point of time ?
Code Snippet:-
HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
try{
IMap map = client.getMap("customers");
map.lock( key );
//business logic
} finally {
map.unlock( key );
}
}
Can you create an mvce and confirm the version of Hazelcast used please.
There are tests for locks here that you can perhaps use as a way to simplify to determine where the fault lies.
Related
Current Situation
we have a MongoDB-Cluster with 1 primary node and 2 secondary nodes
our Spring-Boot application is using the Spring-Data-MongoDB framework to read/write
from/to the cluster
Problem
in some circumstances the MongoDB cluster will change the primary node (for example
during the resizing of the cluster)
this fail-over phase will affect our Spring-Boot application
when some reads or writes are still ongoing and the fail-over happens, we receive an
exception, because the mongoDB-Server is not reachable anymore for our application
we have to deal with this state somehow
Questions
1. What is the best way to handle those faile-over states ?
I've come across the following documentation:
retryable writes
retryable reads
would it be sufficient to set the retryReads and retryWrites flag to true and specify the primary node and the secondary nodes in the connection url? Or should we catch the connection-exception (or alternatively listen to some fail-over-event) and handle those cases by ourself ?
we also have to deal with the following problem: what happens if only 50 % of some bulk-write data got successfully written to the primary node and the other 50 % not ? How handle those cases ideally ?
this leads us to the second question ...
2. How to detect the fail-over event in Spring-Boot ?
for our application a possible solution would be to automatically detect the failover state of the MongoDB-Cluster and than just trigger a restart of our Spring-Boot application.
is there a way to listen to a specific MongoDB-event via spring-data-mongodb in order deal with the case that the primary node has changed?
alternatively: is there a specific exception we should catch and handle?
I hope somebody can help us here.
Thank you in advance!
I've been doing recently some experiments on the behavior of Vert.x and verticles in HA mode. I observed some weaknesses on how Vert.x dispatches the load on various nodes.
1. One node in a cluster crashes
Imagine a configuration with a cluster of some Vert.x nodes (say 4 or 5, 10, whatever), each having some hundreds or thousands verticles. If one node crashes, only one of the remaining nodes will restart all the verticles that had been deployed on the crashed node. Moreover, there is no guarantee that it will be the node with the smallest number of deployed verticles. This is unfair and in worst case, the same node would get all of the verticles from nodes that have crashed before, probably leading to a domino crash scenario.
2. Adding a node to a heavily loaded cluster
Adding a node to a heavily loaded cluster doesn't help to reduce the load on other nodes. Existing verticles are not redistributed on the new node and new verticles are created on the node that invokes the vertx.deployVerticle().
While the first point allows, within some limits, high availability, the second point breaks the promise of simple horizontal scalability.
I may be very possibly wrong: I may have misunderstood something or my configurations are maybe faulty. This question is about confirming this behavior and your advises about how to cope with it or point out my errors. Thanks in for your feedback.
This is how I create my vertx object:
VertxOptions opts = new VertxOptions()
.setHAEnabled(true)
;
// start vertx in cluster mode
Vertx.clusteredVertx(opts, vx_ar -> {
if (vx_ar.failed()) {
...
}
else {
vertx = vertx = vx_ar.result();
...
}
});
and this is how I create my verticles:
DeploymentOptions depOpt = new DeploymentOptions()
.setInstances(1).setConfig(prm).setHa(true);
// deploy the verticle
vertx
.deployVerticle("MyVerticle", depOpt, ar -> {
if(ar.succeeded()) {
...
}
else {
...
}
});
EDIT on 12/25/2019: After reading Alexey's comments, I believe I probably wasn't clear.
By promise of simple horizontal scalability I wasn't meaning that redistributing load upon insertion of a
new node is simple. I was meaning Vert.x promise to the developer that
what he needs to do to have his application to scale horizontally would be
simple. Scale is the very first argument on Vert.x home page, but, you're right, after re-reading carefully there's nothing about horizontal scaling on newly added nodes. I believe I was too much influenced by Elixir or Erlang. Maybe Akka provides this on the JVM, but I didn't try.
Regarding second comment, it's not (only) about the number of requests per second. The load I'm considering here is just the number of verticles "that are doing nothing else that waiting for a message". In a further experiment I can will make this verticle do some work and I will send an update. For the time being, imagine long living verticles that represent in memory actually connected user sessions on a backend. The system runs on 3 (or whatever number) clustered nodes each hosting few thousands (or whatever more) of sessions/verticles. From this state, I added a new node and waited until it is fully integrated in the cluster. Then I killed one of the first 3 nodes. All verticles are restarted fine but only on one node which, moreover, is not guaranteed to be the "empty" one. The destination node seems actually to be chosen at random : I did several tests and I have even observed verticles from all killed nodes being restarted on the same node. On a real platform with sufficient load, that would probably lead to a global crash.
I believe that implementing in Vert.x a fair restart of verticles, ie, distribute the verticles on all remaining nodes based on a given measure of their load (CPU, RAM, #of verticles, ...) would be simpler (not simple) than redistributing the load on a newly inserted node as that would probably require the capability for a scheduler to "steal" verticles from another one.
Yet, on a production system, not being "protected" by some kind of fair distribution of workload on the cluster may lead to big issues and as Vert.x is quite mature I was surprised by the outcome of my experiments, thus thinking I was doing something wrong.
Goal: is to organize blue/green deployment for the particular spring-boot service which is using Kafka.(I'm not interested how to solve B/G on REST or DB layer so lets assume that this part of B/G is already done on LB)
I want: to run two instances of service in docker swarm cluster simultaneously. But one of them must be in sleep mode i.e. do not produce and consume any messages.
Biggest problem: it is not a big deal to set scale = 2 for my service. However in this case each instance of service will consume events and process them. This leads to disaster. So I need simple and transparent mechanism for turning off all services' producers and consumers and restore them with specific offset
I`m looking for an example or suggestions how to achieve that.
Current idea: is to store current offsets in zookeeper and write custom layer that will pool those configs and manage consumers and producers based on that. However, I believe some better and simpler way/framework exists.
There are several approaches:
1. Pause and resume - https://docs.spring.io/spring-kafka/reference/html/_reference.html#pause-resume
For each topic split to 2 topics: active/non-active, and switch between them
If you use public cloud - https://aws.amazon.com/blogs/big-data/best-practices-for-running-apache-kafka-on-aws/
In a service fabric application, using Actors or Services - what would the design be if you wanted to make sure that your block of code would be run on each node.
My first idea would be that it had to be a Service with instance count set to -1, but also in cases that you had set to to 3 instances. How would you make a design where the service ensured that it ran some operation on each instance.
My own idea would be having a Actor with state controlling the operations that need to run, and it would itterate over services using serviceProxy to call methods on each instance - but thats just a naive idea for which I dont know if its possible or if it is the proper way to do so?
Some background info
Only Stateless services can be given a -1 for instance count. You can't use a ServiceProxy to target a specific instance.
Stateful services are deployed using 1 or more partitions (data shards). Partition count is configured in advance, as part of the service deployment and can't be changed automatically. For instance if your cluster is scaled out, partitions aren't added automatically.
Autonomous workers
Maybe you can invert the control flow by running Stateless services (on all nodes) and have them query a 'repository' for work items. The repository could be a Stateful service, that stores work items in a Queue.
This way, adding more instances (scaling out the cluster) increases throughput without code modification. The stateless service instances become autonomous workers.
(opposed to an intelligent orchestrator Actor)
I have a play framework based service that is stateless and intended to be deployed across many machines for horizontal scaling.
This service is handling HTTP JSON requests and responses, and is using CouchDB as its data store again for maximum scalability.
We have a small number of background jobs that need to be run every X seconds across the whole cluster. It is vital that the jobs do not execute concurrently on each machine.
To execute the jobs we're using Actors and the Akka Scheduler (since we're using Scala):
Akka.system().scheduler.schedule(
Duration.create(0, TimeUnit.MILLISECONDS),
Duration.create(10, TimeUnit.SECONDS),
Akka.system().actorOf(LoggingJob.props),
"tick")
(etc)
object LoggingJob {
def props = Props[LoggingJob]
}
class LoggingJob extends UntypedActor {
override def onReceive(message: Any) {
Logger.info("Job executed! " + message.toString())
}
}
Is there:
any built in trickery in Akka/Actors/Play that I've missed that will do this for me?
OR a recognised algorithm that I can put on top of Couchbase (distributed mutex? not quite?) to do this?
I do not want to make any of the instances 'special' as it needs to be very simple to deploy and manage.
Check out Akka's Cluster Singleton Pattern.
For some use cases it is convenient and sometimes also mandatory to
ensure that you have exactly one actor of a certain type running
somewhere in the cluster.
Some examples:
single point of responsibility for certain cluster-wide consistent decisions, or coordination of actions across the cluster system
single entry point to an external system
single master, many workers
centralized naming service, or routing logic
Using a singleton should not be the first design choice. It has
several drawbacks, such as single-point of bottleneck. Single-point of
failure is also a relevant concern, but for some cases this feature
takes care of that by making sure that another singleton instance will
eventually be started.