akka remote actor running on local actor - scala

I am learning akka-remote and trying to re-do http://www.typesafe.com/activator/template/akka-sample-remote-scala myself.
When I try to run the project in two separate JVMs, I see
$ clear;java -jar akkaio-remote/target/akka-remote-jar-with-dependencies.jar com.harit.akkaio.remote.RemoteApp ProcessingActor
ProcessingActorSystem Started
and
$ clear;java -jar akkaio-remote/target/akka-remote-jar-with-dependencies.jar com.harit.akkaio.remote.RemoteApp WatchingActor
WatchingActorSystem Started
asking processor to process
processing big things
I asked my Processing System to run on port 2552
include "common"
akka {
# LISTEN on tcp port 2552
remote.netty.tcp.port = 2552
}
and I told my other system (WatchingSystem) to run on port 2554 but start processingActor on port 2552
include "common"
akka {
actor {
deployment {
"/processingActor/*" {
remote = "akka.tcp://ProcessingActorSystem#127.0.0.1:2552"
}
}
}
remote.netty.tcp.port = 2554
}
and common is about using the right provider
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
netty.tcp {
hostname = "127.0.0.1"
}
}
}
Questions/Concerns
From logs, I see that the processingActor is running on WatchingActorSystem and not on ProcessingActorSystem, what is wrong going on?
How can I see that the two ActorSystems are connecting to each other. I do not see logging happening. However, in the example, I shared the logging happens. What am I missing?
The entire code is posted on Github and runs as well

1) Your deployment configuration is set up to have all the children of processingActor being remote, as described in the akka configuration docs
You should set it to this instead:
deployment {
"/processingActor" {
remote = "akka.tcp://ProcessingActorSystem#127.0.0.1:2552"
}
2) You need to set your log level to something useful as described in the akka logging documentation

Related

Akka Cluster Connection Refused Between Machines

I am attempting to make a project using Akka Clustering, and have been using the akka-cluster-sample-scala from Lightbend(https://github.com/akka/akka-samples/tree/2.6/akka-sample-cluster-scala) as a base. As it lacks much direct information on connecting across a network, I modified the application.conf to look more like this:
akka {
actor {
provider = cluster
serialization-bindings {
"sample.cluster.CborSerializable" = jackson-cbor
}
}
remote {
artery {
canonical.hostname = "127.0.0.1"
canonical.port = 0
}
}
cluster {
seed-nodes = [
"akka://ClusterSystem#131.194.71.132:25251",
"akka://ClusterSystem#131.194.71.132:25252",
"akka://ClusterSystem#131.194.71.133:25251",
"akka://ClusterSystem#131.194.71.133:25252"]
downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
}
}
When run across these two machines, Akka fails to be able to connect over TCP between them, leading to the following warnings:
[info] [2020-09-28 14:34:37,877] [WARN] [akka.stream.Materializer] [] [ClusterSystem-akka.actor.default-dispatcher-5] - [outbound connection to [akka://ClusterSystem#131.194.71.132:25251], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(131.194.71.132:25251,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
Is there anything notably wrong that may be causing this, or something more specifically needing to be reconfigured in order to allow connection over TCP between these machines?
My question was answered by JohanAndren on the Discuss Lightbend forums:
https://discuss.lightbend.com/t/akka-cluster-connection-refused-between-machines/7263
The answer was that you cannot use the 127.0.0.1 localhost definition, all nodes must use their public ips/hostnames and ports.

How do I overwrite config settings from an included file

I have defined some akka remote settings in my application.conf:
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "myhost"
port = 2561
maximum-frame-size = 256000b
}
}
}
But then I have another program that needs to access other configuration settings from my application.conf. But I need to ignore the akka config settings. So I've tried the following for the second program:
include "application"
akka {}
But the akka settings from application.conf are still being applied. I know this because I get a bind exception on the akka port eventhough there should be no remote akka in my second app
What is the best way for me clear/ignore the akka config settings from my application.conf?
Let's say you want to override the akka.remote.netty.tcp.port in your another.conf, you simply
include "application.conf"
akka.remote.netty.tcp.port = 2562
It will override the netty tcp port while leaving the rest unchanged and inherited
After some experimentation, the best bet IMO is to factor out your application.conf so that your common settings are in their own conf files.
So you might put the common settings in a common-settings.conf and then in your Akka application you would have
include "common-settings"
akka {
// Akka settings here
}
And the other modules that need common-settings can just, in their application.conf:
include "common-settings"
This may work better with a multi-module build. If they're the same, e.g., sbt module, then you'll probably replace the canonical application.conf with akka-application.conf and other-application.conf and point your ActorSystem setup code to akka-application.conf instead of application.conf (which probably shouldn't exist in this scenario, as you'd want bare ConfigFactory.load() calls to fail very quickly (the alternative here is to have different programs fighting over who owns application.conf).
The issue you are facing is actually by design. From the HOCON first page documentation:
Duplicate keys are allowed; later values override earlier, except for object-valued keys where the two objects are merged recursively
Therefore, when you add in your second file an akka {} it is just being merged, and not overwritten.
As I can see it, you have 2 options.
Copying the configuration, and override all properties with the one you actually want. That means, that in the second program, you can add:
akka {
actor {
provider = "new value"
warn-about-java-serializer-usage = false
}
remote {
enabled-transports = ["completely new values"]
netty.tcp {
hostname = "etc..."
port = 2561
maximum-frame-size = 256000b
}
}
}
The other option, which I like less, is to overwrite the object called akka. For doing that, you need to assign to it something that is not an object. Otherwise it will just be merged. For instance, if you add the the second program akka=4, so it will completely remove all of the other values. But! In this case, you have your program to deal with those properties to be missing. That means, that somewhere in your code you will have to write something like (don't forget that config throws on missing):
Try(config.getString("akka.actor.provider")).getOrElse(Do something here)
You have to do that because now akka is a string, and you cannot look into that as an object.

Akka Singleton Cluster: resolveOnce of a worker by master fails after restart

I am using Akka Cluster 2.4.3 and trying to setup a simple cluster in my machine to understand its working better. I have a singleton cluster with remoting enabled with primary and standby master and one worker node. Each of these 3 run in separate JVMs
Things work fine when all the nodes are started the first time. If I kill and restart the worker, I see following issues happening
Restart Worker
When the worker comes back after restart, the master on receiving MemberUp event tries to resolve for the actorRef from the member address the following way
context.actorSelection(member.address.toString).resolveOne(15 seconds)
This fails with an exception saying ActorNotFound. This works with no problem when all the nodes are coming up for the first time in the cluster.
Restart worker again
This time, the worker comes up with the following message
[WARN] [04/15/2016 18:24:24.991] [clustersystem-akka.remote.default-remote-dispatcher-5] [akka.remote.Remoting] Tried to associate with unreachable remote address [akka.tcp://clustersystem#host1:2551]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
Restart worker again
This time the resolveOne on a MemberUp event works.
I am having a bit of difficulty in understanding what is happening here, I have looked into the docs but I did not find anything that will help me in there.
application.conf
akka {
actor {
provider = "akka.cluster.ClusterActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
}
log-dead-letters = off
jvm-exit-on-fatal-error = on
loglevel = "DEBUG"
remote {
log-remote-lifecycle-events = off
netty.tcp {
hostname = "host1"
port = 0
}
}
cluster {
seed-nodes = [
"akka.tcp://clustersystem#host1:2551",
"akka.tcp://clustersystem#host1:2552"]
auto-down-unreachable-after = 10s
}
extensions = ["akka.cluster.metrics.ClusterMetricsExtension"]
}
I start master nodes at ports 2551 and 2552 (provide the ports as command line args) and I start the worker on port 3551

Does Akka clustering (2.4) use any ports by default other than 2551?

Does Akka use ports (by default) other than port 2551 for clustering?
I have a 3-node Akka cluster--each node running in a Docker using 2.4's bind-hostname/port. I have a seed running outside a Docker in some test code. I can successfully send messages to the nodes point-to-point, directly, so basic Akka messaging works fine for the Docker-ized nodes.
My Seed code looks like this:
class Seed() extends Actor {
def receive = {
case "report" =>
mediator ! DistributedPubSubMediator.SendToAll("/user/sender", ReportCommand(), false)
case r:ReportCommand => println("Report, please!")
}
}
val seed = system.actorOf(Props(new Seed()),"sender")
val mediator = DistributedPubSub(system).mediator
mediator ! DistributedPubSubMediator.Put(seed)
My worker nodes look like this:
class SenderActor(senderLike:SenderLike) extends Actor {
val mediator = DistributedPubSub(context.system).mediator
mediator ! Put(self)
def receive = {
case report:ReportCommand => println("REPORT CMD!")
}
}
When I run this and send a "report" message to the Seed, I see the Seed's "Report, please!" message, so it received its own broadcast, but the 3 workers in the Dockers don't register having received anything (no output on receive). Not sure what's wrong so I'm wondering if there is another port besides 2551 I need to EXPOSE in my Dockers for clustering?
You'll need to configure Akka using port and bind-port, since in docker the "local port" is different from the port "the outside world can reach me at".
In order to do this see this documentation page: Peer to Peer vs Client Server
And this FAQ section Why are replies not received from a remote actor?

How debug akka association porcess?

Here is a scenario:
I have packaged scala project with spray into jar file.
Launch jar file on RedHat 6.5 on Virtual Box (ip - 192.168.1.38)
Launch jar file on RedHat 6.5 on Virtual Box (ip - 192.168.1.41)
Everything works locally - I can send REST request to each virtual machine and get response.
Problem
Akka systems can not became to cluster. I run 192.168.1.38 with default settings, but 192.168.1.41 have an additional property - akka.cluster.seed-nodes which is set to akka.tcp://mySystem#192.168.1.38:2551. So I get:
[WARN] [12/09/2014 17:10:24.043] [mySystem-akka.remote.default-remote-dispatcher-8] [akka.tcp://mySystem#192.168.1.41:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FmySystem%40192.168.1.38%3A2551-0] Association with remote system [akka.tcp://mySystem#192.168.1.38:2551] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://mySystem#192.168.1.38:2551]].
No other errors or warning. Also how can I test akka association or print debug akka association settings?
Also can linux settings influence to akka association?
Most probably iptables is blocking particular port, if it's your test configuration just disable iptables.
service iptables save
service iptables stop
chkconfig iptables off
service ip6tables save
service ip6tables stop
chkconfig ip6tables off
If it will not help try to check you SELinux configuration using command getenforce and the same for test purposes you can completely disable it. SELinux manual
In case of your application.conf, try using following configuration for each node:
akka {
log-dead-letters = on
loglevel = "debug"
actor
{
provider = "akka.cluster.ClusterActorRefProvider"
}
extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
remote {
log-remote-lifecycle-events = off
netty.tcp {
port = 6001
}
}
cluster {
seed-nodes = [
"akka.tcp://ActorSystem#192.168.1.38:6001",
"akka.tcp://ActorSystem#192.168.1.41:6001"
]
auto-down-unreachable-after = 10s
}
}
All the logs related to the cluster nodes are logged as info but having debug log level in test environment is in general good idea.
When the second, node will join the cluster, you should notice following log:
INFO [ActorSystem-akka.actor.default-dispatcher-4] [Cluster(akka://ActorSystem)] - Cluster Node [akka.tcp://ActorSystem#10.0.1.41:6001] - Marking node(s) as REACHABLE [Member(address = akka.tcp://ActorSystem#10.0.1.41:6001, status = Up)]
Cluster state could be also monitored using jmx akka.Cluster MXBean
{ "self-address": "akka.tcp://ActorSystem#10.0.1.82:6001", "members": [ { "address": "akka.tcp://ActorSystem#10.0.1.82:6001", "status": "Up" } ], "unreachable": [ ] }