Kafka producer.send() is Stopped by producer.close() - scala

Question

I am trying to send the output of a word-count problem (in spark- scala) on a kafka topic named "test". See Below Code:
val Dstream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
val lines = Dstream.map(f => f._2)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.foreachRDD(
rdd => rdd.foreach(
f =>
{
val sendProps = new Properties()
sendProps.put("metadata.broker.list", brokers)
sendProps.put("serializer.class", "kafka.serializer.StringEncoder")
sendProps.put("producer.type", "async")
val config = new ProducerConfig(sendProps)
val producer = new Producer[String, String](config)
producer.send(new KeyedMessage[String, String]"test", f._1 + " " +f._2))
producer.close();
}))
The problem is some words are missing in output randomly. I also noticed that if I removed the statement
producer.close()
there is no data loss.
Does this mean producer.close() interrupts producer.send() before it actually puts data in buffer due to which that particular tuple is not being sent to consumer? If Yes, How shall I close producer without risking data loss?
Above was my initial problem and solved by Vale's answer.
Now, when I change producer.type property again - data goes missing randomly.
sendProps.put("producer.type", "sync")
To clarify producer.send is running for all the words I need to put in output Topic. But, some words go missing and are not displayed in output Kafka Topic.

Answer 1

This is weird. The close() method should wait for the send to have finished, and this was why a close(time) method was introduced: as you can see here.
So, I use Java 7. Is the rdd.foreach operating on each partition inside it? Or is it operating on each Tuple (as I think it's doing)?
If the latter, could you try a rdd.foreachPartition (refer to this)? Because you are creating a producer for each line you take, and I fear this could be causing problems (although theoretically it shouldn't).