I'm trying to publish a simple json to a kafka topic. I'm using the dockerized version of both apache nifi and apache kafka (latest of both). the kafka version is 2.5 but there is no processor to connect to this version.
When I use the other publishKafka processors I get this error:"can not update metadata in 5000 ms".
Can anyone help me about the problem?
Kafka up to 2.6 is now supported in NiFi 1.12.X. You need to upgrade and switch to the new Kafka 2.6 processors.
Related
we have production Kafka cluster with 2.7 version , 5 Kafka brokers on RHEL 7.9 version
we want to upgrade the Kafka version to 3.X version
3.X version not include zookeeper , so we are wondering if we can do upgrade without any data loss
regarding to kafka 2.7 version , Kafka storing the metadata on zookeeper servers ( as brokers ids , topics names etc )
but is it possible to perform rolling upgrade from 2.7 to 3.x version without any data loss?
The upgrade guide should contain all infos you need.
While KRaft mode (without ZooKeeper) is production ready since 3.3, they still keep ZooKeeper around for compatibility until the 4.0 release.
Furthermore If I understand correctly, it is currently only possible to set up a fresh cluster in KRaft mode, but not to migrate an existing one with ZooKeeper. Kafka 3.5 will be a migration version they intend you to migrate from ZooKeeper to KRaft.
This is explained quite nicely in the release notes from Kafka, especially for Kafka 3.3 and the release video
As long as your Kafka brokers are not running with Java 8 still, you can simply do a rolling upgrade from 2.7 to 3.X like you are used to.
I want to add monitoring of my kafka.
I've found product named kafka exporter which depends on 14.9.3. I use kafka 17(https://artifacthub.io/packages/helm/bitnami/kafka/17.0.0) and it doesn't work with latest kafka exporter(https://artifacthub.io/packages/helm/prometheus-community/prometheus-kafka-exporter, version: 1.6.0) because of version incompatibility. I investigated the issue and found out that it is because kafka exporter can't work with kafka 17. I tried to downgrade kafka to to 14.9.3 and problem disappeared.
So my question is: how to monitor Kafka 17 ?
We are planning to upgrade our Kafka client from 2.0.0 to a later version that is be compatible with Kafka broker 2.1.x .
What is the latest Kafka client version that is compatible with Kafka broker 2.1.x?
Is it possible to upgrade from 2.0.0 to 2.3.x?
Is there a compatibility matrix for latest Kafka broker and client versions? Did anyone face any issues with such upgrades?
I have a kafka producer that is using 0.80 Kafka version on one machine (eg. ip is 1.2.3.4), can I use kafka consumer that is using 0.10 Kafka version on another machine to consume the message?
I tried to write a consumer on the newer version which listen to 1.2.3.4:9092. But it says kafka.errors.NoBrokersAvailable: NoBrokersAvailable. Is that not allowed? Or did I set something wrong?
Thanks.
Version 0.10.2 clients can talk to version 0.10.0 or newer brokers. However, if your brokers are older than 0.10.0, you must upgrade all the brokers in the Kafka cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.x and newer clients.
https://kafka.apache.org/documentation/#upgrade_10_2_0
I am using the KafkaConsumer82 and connector jar version is 0.10.2 and kafka version is 0.9.1 and flink version is 1.0.0.
The Java consumer works fine when I run it from with in an IDE as a standalone main program. But when I run it from flink run, then I don't see the messages being consumed and don't see any log in stdout of the JobManager in 1ocalhost:8081. Please let me know what might be the issue.
As a first step I would suggest getting the versions in sync. If you're using Kafka 0.9 and Flink 1.0.0 I would suggest using flink-connector-kafka-0.9 version 1.0.0 which contains FlinkKafkaConsumer09.