Not able to read kafka topic from another Host - apache-kafka

Question

I have two hosts A and B. A has zookeeper and kafka(0.9.0.1) installed. B has spark installed on it.
Now when im trying to read the kafka topic from A through spark streaming and persist the data in to B.
Here is the issue, when i run the spark app, it thoroughly throws the below errors continously,
17/04/20 05:22:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.ConsumerRebalanceFailedException: OlapEventsJamba_appEvent_HOSTA-1492680125896-b52bf576 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:254)
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:156)
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:111)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
17/04/20 05:22:16 WARN TaskSetManager: Lost task 0.0 in stage 8.0 (TID 80, prd-mapr-node1.fishbowl.com): kafka.common.ConsumerRebalanceFailedException: OlapEventsJamba_appEvent_HOSTA-1492680125896-b52bf576 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:254)
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:156)
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:111)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Below are the steps I tried to resolve the issue
set the following values in consumer.props and zookeeper.props
zookeeper.connection.timeout.ms=1000000
zookeeper.session.timeout.ms=10000
rebalance.backoff.ms=10000
Both A and B are telnet'ing and pinging.
Then, I tried to print the schema of the topic message, but the output is only
root
Spark app was creating a _success file in the output directory though.
Here is a snap of the zookeeper logs:
[2017-04-20 09:49:23,648] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x79 zxid:0x1a1f txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:25,978] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0xea zxid:0x1a20 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:28,320] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x15b zxid:0x1a21 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:30,653] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x1cc zxid:0x1a22 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
What can be the issue? Any help is appreciated.