java.lang.IllegalArgumentException kafka console consumer - apache-kafka

Question

We are using Kafka 2.10-0.9.0.2.4.2.0-258 in our environments. We are getting below exception with kafka console consumer on few topics. I am aware that some times messages coming into these topics are too big but they do not exceed message.max.bytes.
./kafka-console-consumer.sh --zookeeper xxx:2181,xxx:2181,xxx:2181 --topic test-topic
{metadata.broker.list=xxx:9092,xxx:9092,xxx:9092, request.timeout.ms=30000, client.id=console-consumer-76015, security.protocol=PLAINTEXT}
[2016-08-28 21:27:54,795] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at kafka.message.Message.sliceDelimited(Message.scala:237)
at kafka.message.Message.key(Message.scala:224)
at kafka.message.MessageAndMetadata.key(MessageAndMetadata.scala:30)
at kafka.consumer.OldConsumer.receive(BaseConsumer.scala:84)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:109)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:69)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:47)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Processed a total of 0 messages
I decreased replica.fetch.max.bytes equal to message.max.bytes and also num.replica.fetchers to 2 as suggested in link below but it did not resolve.
https://issues.apache.org/jira/browse/KAFKA-1196
Any idea what else I should do to make it work?
Any help would be appreciated.
Thanks in advance.

Answer 1

I was having the exact same issue. The root cause was the incompatibility of the kafka jar file between the one that your kafka installation use and the one that you used to develop and run your producer. You can find which version of kafka jars your installation is using in /usr/hdp/current/kafka-broker/libs
In my case, my kafka installation is using kafka_2.10-0.9.0.2.4.2.0-258.jar, but the kafka jar that I bundled with my producer was 0.10.0.1. Once, I switched to 0.9.0.2.4.2.0-258, it worked.
If your cluster is HDP and you are using maven to build your producer, you can find all jar dependencies here http://repo.hortonworks.com/content/repositories/releases/
For maven here is what you have to use:
Repository:
<repositories>
<repository>
<id>org.hortonworks</id>
<url>http://repo.hortonworks.com/content/repositories/releases/</url>
</repository>
</repositories>
Dependency:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.9.0.2.4.2.0-258</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>jmxri</artifactId>
<groupId>com.sun.jmx</groupId>
</exclusion>
<exclusion>
<artifactId>jms</artifactId>
<groupId>javax.jms</groupId>
</exclusion>
<exclusion>
<artifactId>jmxtools</artifactId>
<groupId>com.sun.jdmk</groupId>
</exclusion>
</exclusions>
</dependency>

Answer 2

I had the same issue (error/exception) when running the consumer. However, the scenario was slightly different.
I initially was working with Kafka Version - 2.11-0.10.2.0 (OLD) and then I had to change to 2.10-0.9.0.1 (NEW).
So when I downloaded the NEW setup and just started the zookeeper, broker, producer and finally the consumer I got the above error. I was using the default producer and consumer scripts given in the download itself and testing the quickstart guide.
I got the same error as above. And I could not figure out how the producer inside the freshly downloaded pack could be using a different jar as explained in the answer above.
So, I realized that there must be a common place which is referenced by all the Kafka instance which I found to be the /tmp/kafka-logs and /tmp/zookeeper folders.
Once, deleted them and restarted my NEW download Kafka instance, I was able to get rid of the above exception and things worked smoothly.
Hope this adds to help someone else also facing the same error but in a different scenario.
Shabir