This feels like an obvious question but the answer alludes me. How do I indicate that I want an instance of a stateful service on all nodes in a cluster? With stateless services it's as simple as using -1 for instance count, but that is not supported for stateful.
I'm moving my existing stateless webapi service to stateful so that I can replace my usage of a Redis cache to a reliable dictionary. Because it's a service visible externally and in front of an azure load balancer I need the service running on all instances.
What am I missing?
Stateless instances can have -1 because they can be created and destroyed without impacting state.
The number of partitions for a Stateful Service is not dynamic. It determines how data is sharded across your nodes. Once you choose to have X partitions, you cannot change that number without having to handle the implications to the state yourself.
Alternative approach:
You could add a Stateless Service (as gateway), run that on all nodes, and call a partition of a statefull service from them.
Partitioning the Stateful Service will increase scalability. You could partition for instance on user group/tenant/subscriptionlevel, etc.
Since it is stateful, only one node/VM will be assigned a primary in the cluster and two secondary replicas. The primary will maintain the read/write to the reliable dictionaries/queues and updates the replicas (transaction). This is how it maintains reliability, availability and durability of data. From your stateful services, you can then call your stateless services to process those data. And you can have all the stateles services (deployed in all the nodes/VM in your cluster) execute/process those data.
Related
I have a use case where I need to consume a message in all the instances of service. let's say if my service is running on 5 instances, then the message coming through Kafka needs to be processed on every instance. Since this data is being used in many other APIs so we are storing this in local memory to serve APIs.
Since this data is used very frequently, I don't want to store this data in Redis or some other global cache which will increase latency and cost of network calls.
I want to create a pipeline where any change in data by third-party service will be updated to all the instances and new data is being served in the APIs by all the instances.
It isn't possible with kafka.
It seems that kafka isn't the right choice for this case.
I can suggest 3 solutions:
You can use Redis as you mentioned above, trading off a
little latency.
If the services are running on the same machine you could use a shard memory for all the processes to read from (and then you are agnostic to the process that got the event)
You can hack something but it is an anti-pattern and I won't suggest you to do so as you will probably affect the abilities of the Consumer Group. It's a totally abuse of kafka.
The hack you can do is to consume with a different Consumer Group at each instance. (Let's say a random UUID when you start polling).
Let's assume there is a single consumer group (from kafka perspective). Consumer group consists of 20 replicas of Service instances. All work is balanced among those 20 instances based on some property (UUID). Each instance manages its own storage/state/read which in turn contains only data belonging to that shard only. So there are 20 separate storages, one for each replica. But what happens in case of scaling up or down those Services? How would the remaining 10 Services manage to get all that data previously belonging to other instances? I assume that each service may emit so-called "state event" (stream-table duality?) and other instance may get the responsibility of managing a new part of overall data based on such stream. But this is still a lot of work to do. Such a stream may consist of millions of items (even if compacted). There must be a more efficient way to achieve this. And what if we scale up? Group leader must now inform somehow respective instance to drop part of its data. I have read some books/posts about that matter but I couldn't find any concrete information on how this is managed.
Unclear why this is tagged apache-kafka, since sharding isn't a Kafka term. In Kafka Streams, it can handle distribution of state stores across separate instances using the KTable API. When instances are scaled up and down, the data becomes temporarily unaccessible while the state is rebuilt. Different instances can query each other with "Interactive Queries".
Let's say I have Hello-Service. In Lagom, this service can run across multiple nodes of a single cluster.
So within Cluster 1, we can have multiple "copies" of Hello-Service:
Cluster1: Hello-Service-1, Hello-Service-2, Hello-Service-3
But is it possible to run service Hello-Service across multiple clusters?
Like this:
Cluster1: Hello-Service-1, Hello-Service-2, Hello-Service-3,
Cluster2: Hello-Service-4, Hello-Service-5, Hello-Service-6
What I want to achieve is better scalability of the read-side processors and event consumers:
In Lagom, we need to set up front the number of shards of given event tag within the cluster.
So I wonder if I can just add another cluster to distribute the load across them.
And, of course, I'd like to shard persistent entities by some key.
(Let's say that I'm building a multi-tenant application, I would shard entities by organization id, so all entities of some set of organizations would go into Cluster 1, and entities of another set of organizations would go into Cluster 2, so I can have sharded read side processors per each cluster which handle only subset of events/entities within the cluster (for better scalability)).
With a single cluster approach, as a system grows, a sharded processor within a single cluster may become slower and slower because it needs to handle more and more events.
So as the system grows, I would just add a new cluster (Let's say, Cluster 2, then Cluster 3, which would handle their own subset of events/entities)
If you are using sharded read sides, Lagom will distribute the processing of the shards across all the nodes in the cluster. So, if you have 10 shards, and 6 nodes in 1 cluster, then each node will process between 1-2 shards. If you try to deploy two clusters, 3 nodes each, then you'll end up each node processing 3-4 shards, but every event will be processed twice, once in each cluster. That's not helping scalability, that's doing twice as much work as needs to be done. So I don't see why you would want two clusters, just have one cluster, and the Lagom will distribute the shards evenly across it.
If you are not using sharded read sides, then it doesn't matter how many nodes you have in your cluster, all events will be processed by one node. If you deploy a second cluster, it won't share the load, it will also process the same events, so you'll get double processing of each event by each cluster, which is not what you want.
So, just use sharded read sides, and let Lagom distribute the work across your single cluster for you, that's what it's designed to do.
I have Azure Service Fabric stateless service which doesn't use any endpoints. It takes message from queue, processing it and save results to db.
I want to deploy 10 instances on my 2 nodes. By default I have -1 instancecount - it means that there will be 2 instances for 2 nodes. I can specify instancecount as 1 or 2 and it will be OK, but I cannot set 10 instances, it gives me error.
So I decide to create another instance of my application type. Is it right solution? Is there more elegant way to do this?
There are a few ways you can currently choose from:
Multiple Application instances (as you're doing). Multiple Applications hosting a service. Increases complexity, because instances must be managed.
Multiple Services. One application, hosting multiple services. Same downside as #1.
Multiple Partitions within a service (instead of one SingletonPartition). Downside of this, is that this number is 'fixed'. Changes require redeployment of the type with some downtime.
Use multiple receivers inside one service. Probably a good option, because it gives the least overhead, as creating multiple processes (#1, 2 & 3) creates some overhead.
(an important question is: do you really require multiple instances?)
More info here.
There's a worker dial-in pattern described for Akka, particularly here: http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2. It describes a way to fairly spread a load between multiple remote workers. It assumes there's only one master, and workers discover and register with it. Is there a way to support multiple masters with worker dial-in pattern, which supports fair and deterministic sharing of workers between multiple masters?
I imagine the following situation. Let's say there's a cluster with 2 different node roles: front-end and worker. There are multiple front-end nodes which run HTTP servers. Those front-ends delegate the business logic to actors running on worker nodes. The front-ends are behind simple HTTP round-robin load balancer (Nginx).
I'd like to have a shared pool of worker nodes that can be used by any of the front-ends. If one node has more load than other, it should consume more worker nodes' capacity. If the load is too heavy, I should be able to add more worker nodes (probably automatically via auto-scaling), and they should, again, support all of the front-ends fairly, on a need basis.
There is a couple of naive implementation leading to different deficiencies. If workers somehow decide which single front-end to support, then worker capacity might not be spread fairly, because front-end load is highly dynamic. Alternatively, if workers will register with all of the front-ends, there might be a race condition when multiple front-ends request some work from a single worker. All in all, I don't see a good way of supporting this. Has anyone any better idea?
By using clusters current state we can add more than one master
.match(CurrentClusterState.class, state -> {
for (Member member : state.getMembers()) {
if (member.status().equals(MemberStatus.up())) {
register(member);
}
}
})