Kafka start-up fails with number format exception - apache-kafka

Question

My Kafka start up fails with below error. Look like failure happened while restoring the previous state. One way I can think of resolving is to delete the content of data directory as my data is not important at the moment but what if something like this happens in prod environment? Is this normal?
[2020-09-21 10:17:33,381] INFO [Log partition=__consumer_offsets-15, dir=/Users/jigarnaik/Documents/confluent-5.5.1/data/kafka] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2020-09-21 10:17:33,381] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NumberFormatException: For input string: "00000000000000000000 2"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.base/java.lang.Long.parseLong(Long.java:692)
at java.base/java.lang.Long.parseLong(Long.java:817)
at scala.collection.immutable.StringLike.toLong(StringLike.scala:309)
at scala.collection.immutable.StringLike.toLong$(StringLike.scala:309)
at scala.collection.immutable.StringOps.toLong(StringOps.scala:33)
at kafka.log.Log$.offsetFromFileName(Log.scala:2825)
at kafka.log.Log$.offsetFromFile(Log.scala:2829)
at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:645)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
at kafka.log.Log.loadSegmentFiles(Log.scala:642)
at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:753)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2544)
at kafka.log.Log.loadSegments(Log.scala:747)
at kafka.log.Log.<init>(Log.scala:334)
at kafka.log.MergedLog$.apply(MergedLog.scala:757)
at kafka.log.LogManager.loadLog(LogManager.scala:308)
at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:378)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:66)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Answer 1

The above will happen if you have a logfile with an invalid name. The distinguishing feature here is seeing "kafka.log.Log$.offsetFromFileName" in the stack trace.
In this case you had a file named "00000000000000000000 2.log" in some partition's log directory... not sure how it ended up with a space in the filename but that'll need to be fixed.
Often you can identify which partition it was by looking at the log lines immediately preceding the error; here I'd start by looking in
/Users/jigarnaik/Documents/confluent-5.5.1/data/kafka/__consumer_offsets-15
but that's not always reliable; you may need to look in ALL of your log directories to find the misnamed file.