Just for discussion here,
I was thinking AKKA is great for writing distributed systems but however if your Supervisor and Actors are all in one machine your distributed system will not be highly available. If the machine goes down the whole distributed system goes down with it.
So how about i put the Supervisor in one machine and all the Actors in separate machines. So if one Actor dies there are still others to handle the work. If i bring up a replacement machine. How can the Supervisor know that there is this new machine that can house a new Actor?
Ultimately the Supervisor tree leads to a Root Supervisor. What if the machine that houses the Root Supervisor dies? Does this make it the weakest link in the whole distributed system? How about having an additional Root Supervisor node that one can fail over to? How about having several and have a load balancer in front of all the Root Supervisor to distribute the load?
Some problems are reoccurring and are solved by various cluster tools in Akka, that are all built on the core cluster APIs.
Cluster singleton allow you to have a single instance of an actor in an entire cluster, if the node is downed, the singleton actors will start on a new node. See docs here: http://doc.akka.io/docs/akka/current/scala/cluster-singleton.html#cluster-singleton
Sharding, as mentioned by László is for when you have a high number of actors that you want to maintain one of in the cluster. Docs here:
http://doc.akka.io/docs/akka/current/scala/cluster-sharding.html
Both of these nicely fits together with Akka Persistence, to make an actor arrive at the same state as a previous actor instance on another node.
If you want your actors to be available at all times, you cannot achieve that at the same time as "having only one" (essentially the consistency vs availability problem), but you can use Akka Distributed Data for always available but eventually consistent state. http://doc.akka.io/docs/akka/current/scala/distributed-data.html
You could of course also build your own tools using Actors and the Cluster APIs if none of the existing fits your use case (I wouldn't say it is trivial though ;) ).
First off, there is quite some documentation on the general topic of distributing Actor systems over multiple machines, communicating over the network:
http://doc.akka.io/docs/akka/snapshot/scala/index-network.html
That said, while it is possible to remotely supervise an Actor, you correctly surmised that it does not yield much additional resilience. Rather, use Cluster Sharding to ensure that copies of workers exist on multiple machines, and use a simple cluster-aware ActorRef to send message to them from another machine (without a supervision relationship).
Related
I am new to Alpakka and am considering using it for system integration. What would be the ideal way to maintain the state of the Akka Streams sources across application restarts ?
For example: let's assume I'm using something as follows to continuously read some input data and dump it somewhere. What if it runs for like 4h, then the full JVM crashes and restarts (e.g. k8s restarts my pod or so):
someSource
.via(someTransformation)
.via(someOtherTransformation)
.toMap(...)
.run()
I understand that if someSource is a Kafka source or Kinesis source or some other stateful source, they can keep track of their offset or checkpoint and restart more or less where they left off.
However, many other sources have no such concept, e.g. the Cassandra source, the File source or the RDBMs source. For example, if I shutdown and restart the code provided in the rdms example, it will restart from the top each time.
Am I understanding correctly that there is no mechanism to address that out of the box, s.t. we have to handle it manually ? I would have imagined that this feature would be desired so commonly that it would be handled somehow. If not, how do people typically address that ? Do you use Akka persistence to store some cursors in a few actors? Or do you store the origin offset together with the output data and re-read it on startup?
Or am I looking at all this the wrong way?
It is a feature that is extremely commonly desired, for the reason you suggest.
However, the only generic, reliable way to implement this would be using akka persistence which is probably the single heaviest (e.g. it requires choosing a database) dependency in the Akka ecosystem. Beyond that, it's going to be somewhat source specific. Some (e.g. Kafka, Kinesis) have a means of doing this that's going to fit the bill in nearly every scenario, but for the others, the details of how to store the state of consumption are something on which there will be a lot of differences of opinion. Akka and Alpakka in general tend to shy away from opinionation.
There's a worker dial-in pattern described for Akka, particularly here: http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2. It describes a way to fairly spread a load between multiple remote workers. It assumes there's only one master, and workers discover and register with it. Is there a way to support multiple masters with worker dial-in pattern, which supports fair and deterministic sharing of workers between multiple masters?
I imagine the following situation. Let's say there's a cluster with 2 different node roles: front-end and worker. There are multiple front-end nodes which run HTTP servers. Those front-ends delegate the business logic to actors running on worker nodes. The front-ends are behind simple HTTP round-robin load balancer (Nginx).
I'd like to have a shared pool of worker nodes that can be used by any of the front-ends. If one node has more load than other, it should consume more worker nodes' capacity. If the load is too heavy, I should be able to add more worker nodes (probably automatically via auto-scaling), and they should, again, support all of the front-ends fairly, on a need basis.
There is a couple of naive implementation leading to different deficiencies. If workers somehow decide which single front-end to support, then worker capacity might not be spread fairly, because front-end load is highly dynamic. Alternatively, if workers will register with all of the front-ends, there might be a race condition when multiple front-ends request some work from a single worker. All in all, I don't see a good way of supporting this. Has anyone any better idea?
By using clusters current state we can add more than one master
.match(CurrentClusterState.class, state -> {
for (Member member : state.getMembers()) {
if (member.status().equals(MemberStatus.up())) {
register(member);
}
}
})
I've got some run-time data I'd like to exist on a designated actor on every node in my Akka cluster, which could be updated via internal event or API call to a single node. I could store this data in a shared database to make it permanent, but I'd rather just store it in memory for speed, since it doesn't need to be persisted. Akka Cluster Singleton, Distributed Pub Sub, and possibly other built-in modules use gossip protocols to keep distributed state in sync.
Is there a ready-built way to adopt data synchronization of my own actors across my cluster?
I've thought about just publishing changes to Distributed Pub Sub, but it seems like this wouldn't be resilient to dropped messages. If I stored it in a cluster singleton, it wouldn't be survivable if that node went down. I don't need persistence if the entire cluster goes down, but I do want resilience if individual nodes do.
You should have a look at Akka Distributed Data, which should really be called "Akka Replicated Data", as it will replicate the data across all nodes.
It provides a simple key-value store, and any changes made on one node will be replicated to all others. As all data is kept on all nodes, it's best used for small data sets. Also, the values in your key-value pairs need to be CRDTs (conflict free replicated data types). The module comes with some pre-defined CRDTs that cover a lot of use cases.
What does it mean practically "create one per logical application"? I have an enterprise application in Scala with 5 modules that will be deployed independently. I have used ActorSystem.create("...") to create some 4 or 5 system Actors in each modules like Messaging, Financial, Sales, Workflow, Security.
Do I have to do ActorSystem.create("...") only once? for my enterprise application with 5 modules as above.
Or am I doing it correctly?
It practically means that if you can reuse same thread-pools, akka-system configuration, dead-letters, namespace for actors, event buses - it's better to use one actor system.
So, in your case, module - is the logical application. Some frameworks like OSGi may allow several logical modules to live inside one JVM (physical application), that's probably why "logical application" term was used. However, in most cases (like, I suppose, yours) they are equal - I would recommend you to use one ActorSystem per module.
More generally, tha case of several logical applications inside one physical is some meta-container (like servlet-container), that runs inside one JVM but manages several independent applications (like several deployed .wars) living in the same JVM.
Btw, if you want to manage JVM resources correctly - you can just assign different dispatchers (and maybe thread pools) into different logical groups of actors, and still use one actor-system. So the rule is - if you can use one ActorSystem - just use one. Entities must not be multiplied beyond necessity
P.S. You should also be aware of lookup problem when using multiple actor-systems in one physical application. So if solution proposed there seems like workaround for your architecture - it's also a sign to merge systems together.
There is no right or wrong size here, or a magic formula to do it right.
It depends on the things you want you ActorSystem(s) to achieve and how the application parts relate to each other.
You should separate ActorSystems when they behave largely differenting performance and reliability needs and when the systems behave differently (blocking/ non blocking for example).
A good example would be a typical WebApplication with a Database: The application handling requests could be non blocking (like for example play), the database driver could be blocking (like slick in the old times).
So here it would be a good idea to use separated ActorSystems, to still be able to handle requests to inform the user that the dataabse communication is down.
As everything each ActorSystem comes with a cost, so you should only do it if you need it.
As #dk14 and #Andreas have already said an ActorSystem allows you to share resources ( thread-pools, akka-system configuration, dead-letters, namespace for actors, event buses).
From a sharing perspective it makes sense to have one ActorSystem per JVM and have different dispatchers per logical module. To get the most out of the your Akka actors it is critical that you tune your dispatcher settings to match 1) your application workload 2) your hardware settings (# of cores). For example, if you have some actors doing network IO they should have their own dedicated dispatchers.
You should also consider carefully how many JVMs you want to run on a physical node. For example, if you have a host with 256/512 GB of RAM running a single JVM may not be the best configuration. On the other hand, a physical/VM having 64 GB of RAM will do fine with just one JVM instance
If you Google "what does Akka do", the typical sales pitches you get is that it helps your program scale "up" and/or scale "out". But just like the buzzword "cloud" does nothing to explain the virtualization technologies that comprise a cloud service, I see "scale up/out" as equally-vague buzzwords that probably don't do Akka any real justice.
So let's say I've got a batch processing system full of 100 different types of tasks. Task 1 - 100 are kicking off all day long, doing their thing, whatever it is that they do. How exactly might Akka help me batch system scale "up"? How might it help my system scale "out"?
It scales "out" because it allows you to design and organize cluster of servers. Being message-passing-based, it is pretty much a one-to-one representation of the actual world (machines connected via the network and sending messages to each other). No magic here, it's just that the paradigm of the framework makes it easier to reason about your infrastructure.
It scales "up" because if you buy better hardware it will transparently take advantage of the newly added cores/cpus without you having to change anything.
(When it comes to the Typesafe stack, get used to the buzzword! :) )
Edit after first comment:
You could organize your cluster the way you want :)
Dividing by type/responsibility seems like a good option yes. You could have VM1 with Task1Actor instances, VM2 with Task2Actor instances and if you notice that task 1 is the bottleneck start VM1-bis to add more instances for example.
Since Akka abstracts the whole process of sending/receiving message you can have several JVMs on the same machine, several VMs on the same physical machine, several actual machines, several actual machines with several VMs with several JVMs. You get the idea.
For the Typesafe stack: http://typesafe.com/platform