Error while running kafka producer
./kafka-console-producer.sh --broker-list localhost:9092 --topic testing
Exception in thread "main" java.lang.NoSuchMethodError: kafka.utils.CommandLineUtils$.parseKeyValueArgs(Lscala/collection/Iterable;)Ljava/util/Properties;
at kafka.tools.ConsoleProducer$ProducerConfig.<init>(ConsoleProducer.scala:245)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:35)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
This kind of error is usually related to mismatched versions of Kafka jars. If this is the case, resetting your CLASSPATH should do the trick:
export CLASSPATH=""
Looks like you either have conflicting jars in your classpath or you have mismatched versions of kafka broker and kafka client.
Related
I created some dummydata as a Stream in KSQLDB with
VALUE_FORMAT='JSON' TOPIC='MYTOPIC'
The Setup is over Docker-compose. I am running a Kafka Broker, Schema-registry, ksqldbcli, ksqldb-server, zookeeper
Now I want to consume these records from the topic.
My first and last approach was over the commandline with following command
docker run --net=host --rm confluentinc/cp-schema-registry:5.0.0 kafka-avro-console-consumer
--bootstrap-server localhost:29092 --topic DXT --from-beginning --max-messages 10
--property print.key=true --property print.value=true
--value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer
--key-deserializer org.apache.kafka.common.serialization.StringDeserializer
But that just returns the error
[2021-04-22 21:45:42,926] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:76)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
I also tried it with different use cases in Java Spring but with no prevail. I just cannot consume the created topics.
If I would need to define my own schema, where should I do that and what would be the easiest way because I just created a stream in Ksqldb?
Is there an easy to follow example. I did not specifiy anything else when I created the stream like in the quickstart example on Ksqldb.io. (I added the schema-registry in my deployment)
As I am a noob that is sitting here for almost 10 hours any help would be appreciated.
Edit: I found that pure JSON does not need the Schema-registry with ksqldb. Here.
But how to deserialize it?
If you've written JSON data to the topic then you can read it with the kafka-console-consumer.
The error you're getting (Error deserializing Avro message for id -1…Unknown magic byte!) is because you're using the kafka-avro-console-consumer which attempts to deserialise the topic data as Avro - which it isn't, hence the error.
You can also use PRINT DXT; from within ksqlDB.
Using Lagom 1.4.11, kafka 0.11
The communication to kafka seems to be working, because the producers had produced something.
Only consumer has errors.
[WARN] [03/13/2019 19:36:45.937] [gui-akka.kafka.default-dispatcher-8] [akka://gui/system/kafka-consumer-1] Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[WARN] [03/13/2019 19:36:50.080] [gui-akka.kafka.default-dispatcher-10] [akka://gui/system/kafka-consumer-1] Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
it seemes that my µService has no connection to the kafka.
I hava a kafka-0 pod, kafka-zookeeper-0.
I installed it with
helm install --name kafka --namespace avalon bitnami/kafka -f values.yaml
Is their someone who can help.
Thank you.
added.
I found out that the producer created the topics:
I have no name!#kafka-0:/$ kafka-topics.sh --zookeeper 10.***.21.50:2181 --list
item-issued-topic
topic-item-created
I have no name!#kafka-0:/$ kafka-console-consumer.sh --bootstrap-server kafka-0:9092 --topic topic-item-created --from-beginning
^CProcessed a total of 0 messages
It seemes that kafka does not store the messages.
I also encountered the same problem, which was solved by looking at this document
./kafka-console-consumer.sh --bootstrap-server kafka-com:9094 --topic test --from-beginning --partition 0
Consumer not receiving messages
I’m having an issue with using the Kafka command line tools to get consumer lag for a given group/topic. Currently, I'm trying to use kafka-consumer-groups.sh as mentioned in countless online resources. The following command works just fine: kafka-consumer-groups.sh --bootstrap-server $BROKERS --list
However, if I use kafka-consumer-groups.sh --bootstrap-server $BROKERS --describe --group $GROUP, I get the following output and error:
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
[2018-07-24 00:19:09,139] ERROR admin-client-network-thread exited (kafka.admin.AdminClient) java.lang.NullPointerException
at org.apache.kafka.common.utils.Utils.join(Utils.java:399)
at org.apache.kafka.common.requests.OffsetFetchRequest$Builder.toString(OffsetFetchRequest.java:74)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.apache.kafka.clients.ClientRequest.toString(ClientRequest.java:63)
at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:374)
at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:332)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:409)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199)
at kafka.admin.AdminClient$$anon$1.run(AdminClient.scala:61)
at java.lang.Thread.run(Thread.java:748) Error: Executing consumer group command failed due to The server experienced an unexpected error when processing the request
I've tried setting GROUP to each and every group available and I've used KafkaTool to confirm that these groups exist and are working properly (consuming messages from various topics). I've tried placing strings in the command directly instead of using an environment variable.
Why am I getting this error and what else can I do to debug?
I found something very weird with Kafka.
I have a producer with 3 brokers :
bin/kafka-console-producer.sh --broker-list localhost:9093, localhost:9094, localhost:9095 --topic topic
Then I try to run a consumer with the new API :
bin/kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094,localhost:9095 --topic topic --from-beginning
I got nothing ! BUT if I use the old API :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic topic
I got my messages !
What is wrong with me ?
PS : I am using Kafka 10
I eventually resolved my problem thanks to this similar post : Kafka bootstrap-servers vs zookeeper in kafka-console-consumer
I believe it is a bug / wrong configuration of mine leading to a problem with zookeeper and kafka.
SOLUTION :
First be sure to have enable topic deleting in server.properties files of your brokers :
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true
Then delete the topic :
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic myTopic
Remove all the /tmp/log.dir directories of your brokers.
EDIT : I faced again the problem and I had to remove also the log files of zookeeper in /tmp/zookeeper/version-2/.
Finally delete the topic in /brokers/topics in zookeeper as follow :
$ kafka/bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
rmr /broker/topics/mytopic
And restart your brokers and create your topic again.
After fighting a while with same problem. Specify --partition and console consumer new API works (but hangs..). I have CDH 5.12 + Kafka 0.11 (from parcel).
UPD:
Also find out that Kafka 0.11 (versioned as 3.0.0 in CDH parclel) does not work correctly with consuming messages. After downgrading to Kafka 0.10 it become OK. --partition does not need any more.
I had the same problem, but I was using a single broker instance for my "cluster", and I was getting this error:
/var/log/messages
[2018-04-04 22:29:39,854] ERROR [KafkaApi-20] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
I just added in my server configuration file the setting offsets.topic.replication.factor=1 and restarted. It started to work fine.
I have setup a multi node setup for kafka, everything seems to work well and show no error logs unless i try to push message to one producer. I get a message:
Bootstrap broker host2:2181 disconnected (org.apache.kafka.clients.NetworkClient)
and on the zookeeper logs i am getting:
"WARN Exception causing close of session 0x0 due to java.io.IOException:
Unreasonable length = 1701969920 (org.apache.zookeeper.server.NIOServerCnxn)"
i cleaned up my data directory which is "/var/zookeeper/data" still no luck.
Any help on the the would be much appriciated
Vaibhav looking at this line (Bootstrap broker host2:2181) looks like you are trying to connect to zookeeper instance rather than broker instance. By Default Kafka broker runs on 9092 port. So producer and consumer should be created as per below command
Producer :
bin/kafka-console-producer.sh --broker-list host1:9092,host2:9092 \
--topic "topic_name"
Consumer:
bin/kafka-console-consumer.sh --bootstrap-server <host_ip_of_producer>:9092 \
--topic "topic_name" --from-beginning