I am running the example provided by official akka: https://github.com/akka/akka-samples/tree/2.5/akka-sample-cluster-scala.
My OS is: Linux Mint 19 with the latest kernel.
And for the Worker Dial-in Example(Transformation Example), I cannot fully run this example as there is no enough space in /dev/shm. Although I have more than 2GB available space.
The problem is when I launch the first frontend node, it eats some KBs space. When I launch the second one, it eats some MBs space. When I launch the third one, it eats some hundred of MBs space. Further I just cannot even launch the fourth one, it just throws an error which causes the whole cluster down:
[info] Warning: space is running low in /dev/shm (tmpfs) threshold=167,772,160 usable=95,424,512
[info] Warning: space is running low in /dev/shm (tmpfs) threshold=167,772,160 usable=45,088,768
[info] [ERROR] [11/05/2018 21:03:56.156] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem#127.0.0.1:57246/] swallowing exception during message send
[info] io.aeron.exceptions.RegistrationException: IllegalStateException : Insufficient usable storage for new log of length=50335744 in /dev/shm (tmpfs)
[info] at io.aeron.ClientConductor.onError(ClientConductor.java:174)
[info] at io.aeron.DriverEventsAdapter.onMessage(DriverEventsAdapter.java:81)
[info] at org.agrona.concurrent.broadcast.CopyBroadcastReceiver.receive(CopyBroadcastReceiver.java:100)
[info] at io.aeron.DriverEventsAdapter.receive(DriverEventsAdapter.java:56)
[info] at io.aeron.ClientConductor.service(ClientConductor.java:660)
[info] at io.aeron.ClientConductor.awaitResponse(ClientConductor.java:696)
[info] at io.aeron.ClientConductor.addPublication(ClientConductor.java:371)
[info] at io.aeron.Aeron.addPublication(Aeron.java:259)
[info] at akka.remote.artery.aeron.AeronSink$$anon$1.<init>(AeronSink.scala:103)
[info] at akka.remote.artery.aeron.AeronSink.createLogicAndMaterializedValue(AeronSink.scala:100)
[info] at akka.stream.impl.GraphStageIsland.materializeAtomic(PhasedFusingActorMaterializer.scala:630)
[info] at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:450)
[info] at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:415)
[info] at akka.stream.impl.PhasedFusingActorMaterializer.materialize(PhasedFusingActorMaterializer.scala:406)
[info] at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:588)
[info] at akka.remote.artery.Association.runOutboundOrdinaryMessagesStream(Association.scala:726)
[info] at akka.remote.artery.Association.runOutboundStreams(Association.scala:657)
[info] at akka.remote.artery.Association.associate(Association.scala:649)
[info] at akka.remote.artery.AssociationRegistry.association(Association.scala:989)
[info] at akka.remote.artery.ArteryTransport.association(ArteryTransport.scala:724)
[info] at akka.remote.artery.ArteryTransport.send(ArteryTransport.scala:710)
[info] at akka.remote.RemoteActorRef.$bang(RemoteActorRefProvider.scala:591)
[info] at akka.actor.ActorRef.tell(ActorRef.scala:124)
[info] at akka.actor.ActorSelection$.rec$1(ActorSelection.scala:265)
[info] at akka.actor.ActorSelection$.deliverSelection(ActorSelection.scala:269)
[info] at akka.actor.ActorSelection.tell(ActorSelection.scala:46)
[info] at akka.actor.ScalaActorSelection.$bang(ActorSelection.scala:280)
[info] at akka.actor.ScalaActorSelection.$bang$(ActorSelection.scala:280)
[info] at akka.actor.ActorSelection$$anon$1.$bang(ActorSelection.scala:198)
[info] at akka.cluster.ClusterCoreDaemon.gossipTo(ClusterDaemon.scala:1330)
[info] at akka.cluster.ClusterCoreDaemon.gossip(ClusterDaemon.scala:1047)
[info] at akka.cluster.ClusterCoreDaemon.gossipTick(ClusterDaemon.scala:1010)
[info] at akka.cluster.ClusterCoreDaemon$$anonfun$initialized$1.applyOrElse(ClusterDaemon.scala:496)
[info] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[info] at akka.actor.Actor.aroundReceive(Actor.scala:517)
[info] at akka.actor.Actor.aroundReceive$(Actor.scala:515)
[info] at akka.cluster.ClusterCoreDaemon.aroundReceive(ClusterDaemon.scala:295)
[info] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
[info] at akka.actor.ActorCell.invoke(ActorCell.scala:557)
[info] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[info] at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[info] at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[info] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
It seems it is sending huge message(48MB+?) to every nodes.
So what's up here? What is the root cause and how shall I fix this?
When I run a topology of Storm with KafkaSpout in Heron, the following exception occurs:
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.HeronInstance:
Starting instance container_2_ads_2 for topology AdvertisingTopology and topologyId AdvertisingTopologyf7b4acbe-bdbc-4772-aaa4-9dd2f113f405 for component ads with taskId 2 and componentIndex 0 and stmgrId stmgr-2 and stmgrPort 31162 and metricsManagerPort 31067
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.HeronInstance: System Config: {heron.streammgr.network.backpressure.lowwatermark.mb=50, heron.streammgr.connection.write.batch.size.mb=1, heron.streammgr.stateful.buffer.size.mb=100, heron.instance.internal.bolt.write.queue.capacity=128, heron.instance.tuning.expected.spout.read.queue.size=512, heron.metricsmgr.network.write.batch.size.bytes=ByteAmount{32768 bytes}, heron.instance.reconnect.streammgr.interval.sec=PT5S, heron.instance.tuning.interval.ms=PT0.1S, heron.instance.emit.batch.size.bytes=ByteAmount{32768 bytes}, heron.logging.directory=log-files, heron.check.tmaster.location.interval.sec=120, heron.instance.reconnect.metricsmgr.interval.sec=PT5S, heron.streammgr.client.reconnect.tmaster.max.attempts=30, heron.streammgr.network.backpressure.highwatermark.mb=100, heron.instance.network.read.batch.size.bytes=ByteAmount{32768 bytes}, heron.instance.tuning.expected.metrics.write.queue.size=8, heron.instance.internal.spout.write.queue.capacity=128, heron.instance.force.exit.timeout.ms=PT2S, heron.tmaster.network.stats.options.maximum.packet.mb=1, heron.streammgr.xormgr.rotatingmap.nbuckets=3, heron.instance.set.control.tuple.capacity=1024, heron.metricsmgr.network.read.batch.size.bytes=ByteAmount{32768 bytes}, heron.streammgr.client.reconnect.tmaster.interval.sec=10, heron.instance.execute.batch.time.ms=PT0.016S, heron.metrics.export.interval.sec=PT1M, heron.streammgr.connection.read.batch.size.mb=1, heron.streammgr.cache.drain.size.mb=100, heron.tmaster.network.master.options.maximum.packet.mb=16, heron.tmaster.establish.retry.interval.sec=1, heron.metrics.max.exceptions.per.message.count=1024, heron.tmaster.stmgr.state.timeout.sec=60, heron.instance.network.write.batch.size.bytes=ByteAmount{32768 bytes}, heron.logging.err.threshold=3, heron.tmaster.network.controller.options.maximum.packet.mb=1, heron.tmaster.metrics.collector.maximum.exception=256, heron.instance.network.write.batch.time.ms=PT0.016S, heron.instance.network.options.socket.send.buffer.size.bytes=ByteAmount{6 MB (6553600 bytes)}, heron.streammgr.mempool.max.message.number=512, heron.logging.maximum.size.mb=100, heron.streammgr.tmaster.heartbeat.interval.sec=10, heron.instance.network.read.batch.time.ms=PT0.016S, heron.tmaster.metrics.network.bindallinterfaces=false, heron.streammgr.network.options.maximum.packet.mb=10, heron.instance.tuning.expected.bolt.write.queue.size=8, heron.metricsmgr.network.options.socket.received.buffer.size.bytes=ByteAmount{8 MB (8738000 bytes)}, heron.logging.maximum.files=5, heron.instance.network.options.socket.received.buffer.size.bytes=ByteAmount{8 MB (8738000 bytes)}, heron.instance.execute.batch.size.bytes=ByteAmount{32768 bytes}, heron.instance.acknowledgement.nbuckets=10, heron.metricsmgr.network.read.batch.time.ms=PT0.016S, heron.metricsmgr.network.options.socket.send.buffer.size.bytes=ByteAmount{6 MB (6553600 bytes)}, heron.metricsmgr.network.options.maximum.packetsize.bytes=ByteAmount{1 MB (1048576 bytes)}, heron.instance.tuning.expected.bolt.read.queue.size=8, heron.logging.flush.interval.sec=10, heron.streammgr.cache.drain.frequency.ms=10, heron.tmaster.establish.retry.times=30, heron.instance.network.options.maximum.packetsize.bytes=ByteAmount{10 MB (10485760 bytes)}, heron.instance.tuning.current.sample.weight=0.8, heron.instance.reconnect.streammgr.times=60, heron.logging.prune.interval.sec=300, heron.instance.reconnect.metricsmgr.times=60, heron.tmaster.metrics.collector.maximum.interval.min=PT3H, heron.tmaster.metrics.collector.purge.interval.sec=PT1M, heron.streammgr.client.reconnect.interval.sec=1, heron.instance.internal.spout.read.queue.capacity=1024, heron.instance.ack.batch.time.ms=PT0.128S, heron.instance.set.data.tuple.size.bytes=ByteAmount{8 MB (8388608 bytes)}, heron.instance.tuning.expected.spout.write.queue.size=8, heron.instance.internal.bolt.read.queue.capacity=128, heron.instance.set.data.tuple.capacity=1024, heron.instance.metrics.system.sample.interval.sec=PT10S, heron.streammgr.network.backpressure.threshold=3, heron.instance.emit.batch.time.ms=PT0.016S, heron.metricsmgr.network.write.batch.time.ms=PT0.016S, heron.instance.internal.metrics.write.queue.capacity=128}
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.HeronClient: Connecting to endpoint: /127.0.0.1:31162
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.HeronClient: Connecting to endpoint: /127.0.0.1:31067
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Connected to Stream Manager. Ready to send register request
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.MetricsManagerClient: Connected to Metrics Manager. Ready to send register request
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Stop writing due to not yet connected to Stream Manager.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Stop writing due to not yet connected to Stream Manager.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: We registered ourselves to the Stream Manager
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Handling assignment message from response
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: We received a new Physical Plan.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Push to Slave
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.MetricsManagerClient: We registered ourselves to the Metrics Manager
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.utils.misc.PhysicalPlanHelper: Building configs for component: ads
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.utils.misc.PhysicalPlanHelper: Added topology-level configs: {topology.acker.executors=2, topology.workers=3, topology.skip.missing.kryo.registrations=false, topology.enable.message.timeouts=true, topology.serializer.classname=org.apache.storm.serialization.HeronPluggableSerializerDelegate, topology.debug=false, topology.max.spout.pending=100, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, topology.fall.back.on.java.serialization=false, topology.name=AdvertisingTopology, topology.component.parallelism=1, topology.stmgrs=3, topology.reliability.mode=ATLEAST_ONCE, topology.message.timeout.secs=30}
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.utils.misc.PhysicalPlanHelper: Added component-specific configs: {topology.acker.executors=2, config.zkRoot=/ad-events/6647e83d-6bd8-454e-ad91-d3ec0a012e62, topology.workers=3, topology.skip.missing.kryo.registrations=false, topology.enable.message.timeouts=true, topology.serializer.classname=org.apache.storm.serialization.HeronPluggableSerializerDelegate, topology.debug=false, topology.max.spout.pending=100, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, topology.fall.back.on.java.serialization=false, topology.name=AdvertisingTopology, topology.component.parallelism=1, config.topics=ad-events, topology.stmgrs=3, topology.reliability.mode=ATLEAST_ONCE, topology.message.timeout.secs=30, config.zkNodeBrokers=/brokers}
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.Slave: Incarnating ourselves as ads with task id 2
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.spout.SpoutInstance: Is this topology stateful: false
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.spout.SpoutInstance: Enable Ack: true
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.spout.SpoutInstance: EnableMessageTimeouts: true
[2018-11-01 22:43:49 +0800] [SEVERE] com.twitter.heron.instance.HeronInstance: Exception caught in thread: SlaveThread with id: 12
java.lang.NullPointerException
at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:80)
at org.apache.storm.topology.IRichSpoutDelegate.open(IRichSpoutDelegate.java:53)
at com.twitter.heron.instance.spout.SpoutInstance.init(SpoutInstance.java:173)
at com.twitter.heron.instance.Slave.startInstanceIfNeeded(Slave.java:222)
at com.twitter.heron.instance.Slave.handleNewAssignment(Slave.java:173)
at com.twitter.heron.instance.Slave.handleNewPhysicalPlan(Slave.java:349)
at com.twitter.heron.instance.Slave.access$300(Slave.java:49)
at com.twitter.heron.instance.Slave$1.run(Slave.java:118)
at com.twitter.heron.common.basics.WakeableLooper.executeTasksOnWakeup(WakeableLooper.java:160)
at com.twitter.heron.common.basics.WakeableLooper.runOnce(WakeableLooper.java:89)
at com.twitter.heron.common.basics.WakeableLooper.loop(WakeableLooper.java:79)
at com.twitter.heron.instance.Slave.run(Slave.java:180)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.HeronInstance: Waiting for process exit in PT2S
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.Slave: Closing the Slave Thread
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.utils.metrics.MetricsCollector: Forcing to gather all metrics and flush out.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.Slave: Shutting down the instance
[2018-11-01 22:43:49 +0800] [WARNING] com.twitter.heron.common.basics.SysUtils: Failed to close com.twitter.heron.instance.Slave#4bef4d93
java.lang.NullPointerException
at org.apache.storm.kafka.KafkaSpout.close(KafkaSpout.java:136)
at org.apache.storm.topology.IRichSpoutDelegate.close(IRichSpoutDelegate.java:58)
at com.twitter.heron.instance.spout.SpoutInstance.clean(SpoutInstance.java:195)
at com.twitter.heron.instance.spout.SpoutInstance.shutdown(SpoutInstance.java:204)
at com.twitter.heron.instance.Slave.close(Slave.java:238)
at com.twitter.heron.common.basics.SysUtils.closeIgnoringExceptions(SysUtils.java:66)
at com.twitter.heron.instance.HeronInstance$SlaveExitTask.run(HeronInstance.java:428)
at com.twitter.heron.instance.HeronInstance$DefaultExceptionHandler.handleException(HeronInstance.java:396)
at com.twitter.heron.instance.HeronInstance$DefaultExceptionHandler.uncaughtException(HeronInstance.java:360)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.instance.Gateway: Closing the Gateway thread
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.utils.metrics.MetricsCollector: Forcing to gather all metrics and flush out.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.MetricsManagerClient: Flushing all pending data in MetricsManagerClient
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: Flushing all pending data in StreamManagerClient
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.SocketChannelHelper: Forcing to flush data to socket with best effort.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.HeronClient: To stop the HeronClient.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.MetricsManagerClient: MetricsManagerClient exits
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.SocketChannelHelper: Forcing to flush data to socket with best effort.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.common.network.HeronClient: To stop the HeronClient.
[2018-11-01 22:43:49 +0800] [INFO] com.twitter.heron.network.StreamManagerClient: StreamManagerClient exits.
[2018-11-01 22:43:49 +0800] [SEVERE] com.twitter.heron.instance.HeronInstance: Instance Process exiting.
And the codes of the topology as follows:
String zkServerHosts = "MY_ZK_IP:2181";
ZkHosts hosts = new ZkHosts(zkServerHosts);
SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopic, "/" + kafkaTopic, UUID.randomUUID().toString());
spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
And the location of the NPE is 80 lines of the open method in KafkaSpout class:
public Object getValueAndReset() {
List<PartitionManager> pms = KafkaSpout.this.coordinator.getMyManagedPartitions();
Set<Partition> latestPartitions = new HashSet();
Iterator var3 = pms.iterator();
PartitionManager pm;
while(var3.hasNext()) { // the line of NPE happened
pm = (PartitionManager)var3.next();
latestPartitions.add(pm.getPartition());
}
this.kafkaOffsetMetric.refreshPartitions(latestPartitions);
var3 = pms.iterator();
while(var3.hasNext()) {
pm = (PartitionManager)var3.next();
this.kafkaOffsetMetric.setOffsetData(pm.getPartition(),
pm.getOffsetData());
}
return this.kafkaOffsetMetric.getValueAndReset();
}
I don't know what caused this problem and how to fix it. Any help is grateful.
NEW EDITED:
All imports have been pointed to the heron-storm classes, but the NPE still happened.
import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.StormSubmitter;
import org.apache.storm.kafka.KafkaSpout;
import org.apache.storm.kafka.SpoutConfig;
import org.apache.storm.kafka.StringScheme;
import org.apache.storm.kafka.ZkHosts;
import org.apache.storm.spout.SchemeAsMultiScheme;
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.base.BaseRichBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
The Storm based Kafka spout does not work with the native Heron topology API. You will need to use the heron-storm API in the compatible mode (add this dependency to your pom file) to build your topology and interface with the Storm-Kafka spout. It should just be a case of swapping the heron imports for heron-storm imports in your bolts.
Some examples of using the heron-storm api are shown here.
Storm and Heron activate their bolts/spouts in different ways, which can cause issues with Storm only code in native Heron topologies.
This is resolved.
final KafkaSpout<byte[], byte[]> spout =
new KafkaSpout<byte[], byte[]>(kafkaSpoutConfig) {
#Override
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
super.open(conf, context, collector);
super.activate();
}
};
I have started kafka-manager on centos VM and below are its logs.
[info] o.a.z.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
[info] o.a.z.ZooKeeper - Client environment:java.io.tmpdir=/tmp
[info] o.a.z.ZooKeeper - Client environment:java.compiler=
[info] o.a.z.ZooKeeper - Client environment:os.name=Linux
[info] o.a.z.ZooKeeper - Client environment:os.arch=amd64
[info] o.a.z.ZooKeeper - Client environment:os.version=3.10.0-862.el7.x86_64
[info] o.a.z.ZooKeeper - Client environment:user.name=root
[info] o.a.z.ZooKeeper - Client environment:user.home=/root
[info] o.a.z.ZooKeeper - Client environment:user.dir=/root/Confluent_kafka/kafka-manager-1.3.3.21
[info] o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:3181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#73687e45
[info] o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:3181. Will not attempt to authenticate using SASL (unknown error)
[info] o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:3181, initiating session
[info] k.m.a.KafkaManagerActor - zk=localhost:3181
[info] k.m.a.KafkaManagerActor - baseZkPath=/kafka-manager
[info] o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:3181, sessionid = 0x16565ff95660000, negotiated timeout = 60000
[info] k.m.a.KafkaManagerActor - Started actor akka://kafka-manager-system/user/kafka-manager
[info] k.m.a.KafkaManagerActor - Starting delete clusters path cache...
[info] k.m.a.DeleteClusterActor - Started actor akka://kafka-manager-system/user/kafka-manager/delete-cluster
[info] k.m.a.DeleteClusterActor - Starting delete clusters path cache...
[info] k.m.a.KafkaManagerActor - Starting kafka manager path cache...
[info] k.m.a.DeleteClusterActor - Adding kafka manager path cache listener...
[info] k.m.a.DeleteClusterActor - Scheduling updater for 10 seconds
[info] k.m.a.KafkaManagerActor - Adding kafka manager path cache listener...
[info] play.api.Play - Application started (Prod)
[info] p.c.s.NettyServer - Listening for HTTP on /0.0.0.0:9000
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Shutting down kafka manager
Tha kafka manager starts perfectly but the WEB UI does not load at all.
The IPV6 is disabled and the netstat shows this
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 2670/java
Can someone help in this.
2016-09-05 19:34:51.079 [info] <0.498.0>#ejabberd_http:init:157 started: {gen_tcp,#Port<0.3947>}
2016-09-05 19:35:06.894 [info] <0.491.0>#ejabberd_listener🉑299 (#Port<0.3948>) Accepted connection 103.255.4.61:40738 -> 162.243.172.244:5222
2016-09-05 19:35:07.306 [info] <0.500.0>#ejabberd_c2s:wait_for_feature_request:789 ({socket_state,gen_tcp,#Port<0.3948>,<0.499.0>}) Failed authentication for 57a6697147e6e21b268b456d#server1.**.com from 193.355.4.61
2016-09-05 19:35:08.225 [info] <0.491.0>#ejabberd_listener🉑299 (#Port<0.3949>) Accepted connection 103.255.4.61:40739 -> 162.243.172.244:5222
2016-09-05 19:35:08.503 [info] <0.502.0>#ejabberd_c2s:wait_for_feature_request:789 ({socket_state,gen_tcp,#Port<0.3949>,<0.501.0>}) Failed authentication for 57a6697147e6e21b268b456d#server1.**.com from 193.355.4.61
Getting this error when XMPP-master client app try to send Request to Driver app
Scala Play app with reactive mongo akka actors
sbt run:
I occasionaly get this error like ten times on sbt console:
It usually happens after a recompile and play reload, otherwise it happens less often, it doesn't cause anything to fail, sometimes page gives error.
[error] r.api.Cursor - fails to send request
java.util.concurrent.TimeoutException: Futures timed out after [3000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[scala-library-2.11.7.jar:na]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153) ~[scala-library-2.11.7.jar:na]
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169) ~[scala-library-2.11.7.jar:na]
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169) ~[scala-library-2.11.7.jar:na]
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169) ~[akka-actor_2.11-2.3.13.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640) [scala-library-2.11.7.jar:na]
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167) ~[akka-actor_2.11-2.3.13.jar:na]
at scala.concurrent.Await$.ready(package.scala:169) ~[scala-library-2.11.7.jar:na]
at reactivemongo.api.DefaultCursor$Impl$$anonfun$awaitFailover$1.apply(cursor.scala:474) ~[reactivemongo_2.11-0.11.10.jar:0.11.10]
at reactivemongo.api.DefaultCursor$Impl$$anonfun$awaitFailover$1.apply(cursor.scala:474) ~[reactivemongo_2.11-0.11.10.jar:0.11.10]
Also When I start the app (connect to reactivemongo driver mongodb) I get this output:
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The node set is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
[info] r.c.actors.MongoDBSystem - The primary is now authenticated
why so much?
how do i debug this kind of stuff.
Edit:
I connect to mongolab.com mongodb so i tried this:
val defaultStrategy = FailoverStrategy()
val customStrategy =
FailoverStrategy(
delayFactor = attempt => 10
)
// database-wide strategy
val db = connection.db("dbname", customStrategy)
but i saw a timeout of 10500! network is fine, omg what is wrong with this reactivemongo?
I have this:
val RM = "org.reactivemongo" %% "reactivemongo" % "0.11.10"
val PRM = "org.reactivemongo" %% "play2-reactivemongo" % "0.11.10"
What does this error mean (it doesn't crash the app or anything why), and what kind of code causes this?
I have the same problem, which I could steadily reproduce while testing.
I tried switching to Reactivemongo fresh release 0.11.11 the exception stopped, so as whole system performance went down.
My final solution is - increase number of connections, and change failover strategy
val mongoDriver = new MongoDriver(Some(configuration.underlying))
Runtime.getRuntime().addShutdownHook(new Thread() {
override def run() = {
mongoDriver.close()
}
})
val connectionOptions = MongoConnectionOptions(
nbChannelsPerNode = 40,
connectTimeoutMS = 5000
)
val mongoConnection: MongoConnection = mongoDriver.connection(List(mongoHost), connectionOptions)
val customStrategy = FailoverStrategy(
retries = 8,
delayFactor = n => n * 1.5
)
mongoConnection(mongoDB, customStrategy)
Now I don't see them, or they are harder to reproduce.