Apache kafka Error while appending records to topic - apache-kafka

Question

I am trying to consume 10 millions rows csv file of size(600MB) via connect api. The connection starts consuption completes for 3.7million records .After that i am getting the below error.
[2018-11-01 07:28:49,889] ERROR Error while appending records to topic-test-0 in dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:95)
at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:151)
at kafka.log.LogSegment.append(LogSegment.scala:138)
at kafka.log.Log.$anonfun$append$2(Log.scala:868)
at kafka.log.Log.maybeHandleIOException(Log.scala:1837)
at kafka.log.Log.append(Log.scala:752)
at kafka.log.Log.appendAsLeader(Log.scala:722)
at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:634)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)
at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:622)
at kafka.server.ReplicaManager.$anonfun$appendToLocalLog$2(ReplicaManager.scala:745)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:138)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:236)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:229)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:138)
at scala.collection.TraversableLike.map(TraversableLike.scala:234)
at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:733)
at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:472)
at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:489)
at kafka.server.KafkaApis.handle(KafkaApis.scala:106)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.lang.Thread.run(Thread.java:748)
[2018-11-01 07:28:49,893] INFO [ReplicaManager broker=0] Stopping serving replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
[2018-11-01 07:28:49,897] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,topic-test-0,__consumer_offsets-25,__consumer_offsets
i have single topic name topic-test
Machine Specs:
OS : CentOs 7
Ram : 16GB
HD : 80GB
i see some blogs talking about log.dirs is server.property but things are not clear as how it wants the input. Also am i to create partition ? I didnot do it thinking it is the same data file.

Answer 1

ERROR Error while appending records to topic-test-0 in dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)java.io.IOException: No space left on device
This comes when you are consuming a huge file or stream in kafka topic.
Go to default log directory /tmp/kafka-logs
Then,
[root#ENT-CL-015243 kafka-logs]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_rhel6u4x64-lv_root 61G 8.4G 49G 15% /
tmpfs 7.7G 0 7.7G 0% /dev/shm
/dev/sda1 485M 37M 423M 9% /boot
/dev/mapper/vg_rhel6u4x64-lv_home 2.0G 68M 1.9G 4% /home
/dev/mapper/vg_rhel6u4x64-lv_tmp 4.0G 315M 3.5G 9% /tmp
/dev/mapper/vg_rhel6u4x64-lv_var 7.9G 252M 7.3G 4% /var
As you can see in my case only 3.5Gb of /tmp space was available and I was facing this issue. I create a /klogs in root and changed log.dirs=/klogs/kafka-logs in server.properties