Spark consumer doesn't read Kafka producer messages Scala - scala

Question

I am trying to create Kafka producer connected to Spark consumer. The producer works fine, however, the consumer in Spark does not read the data from the topic for some reason. I run kafka using spotify/kafka image in docker-compose.
Here is my consumer:
object SparkConsumer {
def main(args: Array[String]) {
val spark = SparkSession
.builder()
.appName("KafkaSparkStreaming")
.master("local[*]")
.getOrCreate()
val ssc = new StreamingContext(spark.sparkContext, Seconds(3))
val topic1 = "topic1"
def kafkaParams = Map[String, Object](
"bootstrap.servers" -> "localhost:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "group1",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val lines = KafkaUtils.createDirectStream[String, String](
ssc,
LocationStrategies.PreferConsistent,
ConsumerStrategies.Subscribe[String, String](Set(topic1), kafkaParams)
)
lines.print()
}
Kafka Producer looks like this:
object KafkaProducer {
def main(args: Array[String]) {
val events = 10
val topic = "topic1"
val brokers = "localhost:9092"
val random = new Random()
val props = new Properties()
props.put("bootstrap.servers", brokers)
props.put("client.id", "KafkaProducerExample")
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
val producer = new KafkaProducer[String, String](props)
val t = System.currentTimeMillis()
for (nEvents <- Range(0, events)) {
val key = null
val values = "2017-11-07 04:06:03"
val data = new ProducerRecord[String, String](topic, key, values)
producer.send(data)
System.out.println("sent : " + data.value())
}
System.out.println("sent per second: " + events * 1000 / (System.currentTimeMillis() - t))
producer.close()
}
}
UPDATE:
My docker-compose file with Kafka:
version: '3.3'
services:
kafka:
image: spotify/kafka
ports:
- "9092:9092"

Answer 1

This is a common problem using Kafka with Docker. First, you should check what is the configuration in zookeeper for your topic. You can use the Zookeeper scripts inside the Kafka container. Probably when your topic is created the ADVERTISED_HOST is the name of your service. So when the consumer tries to connect to the broker, this returns "kafka" as the broker location. Because you are running the consumer outside the docker network, your consumer will never connect to the broker for consuming. Try to set the env for your kafka container with ADVERTISED_HOST=localhost.