I have seen that i can either
(1) Use the ZooKeeper that comes with Kafka, or
(2) Use the ZooKeeper from Apache itself.
Which is the preferred method(if there is one) and why? My use-case is for a small application, so it will be a 3 ZooKeeper ensemble/cluster. I am using Window 10 for my test. The ZooKeeper version i am using is 3.5.6. The Apache Kafka version i am using is 2.12-2.3.0
Note:
I have tried both ways i.e (1) and (2), and both work.
UPDATE:
Found what i was looking for. For use case (2), if i want to use Kafka 3.0.0, ideally, i will use it with Zookeeper 3.6.3 as that's what it has been tested with, as noted here.
ZooKeeper has been upgraded to version 3.6.3.
Kafka is tested against the Zookeeper version it comes with.
If you want to upgrade, you'll need to verify Zookeeper itself is backwards compatible with older clients/protocols that Kafka may use.
Its unlikely that jumping from 3.4.x to 3.5.x is a compatible change, but if you stay within the same minor release, it should be fine
Related
we have production Kafka cluster with 2.7 version , 5 Kafka brokers on RHEL 7.9 version
we want to upgrade the Kafka version to 3.X version
3.X version not include zookeeper , so we are wondering if we can do upgrade without any data loss
regarding to kafka 2.7 version , Kafka storing the metadata on zookeeper servers ( as brokers ids , topics names etc )
but is it possible to perform rolling upgrade from 2.7 to 3.x version without any data loss?
The upgrade guide should contain all infos you need.
While KRaft mode (without ZooKeeper) is production ready since 3.3, they still keep ZooKeeper around for compatibility until the 4.0 release.
Furthermore If I understand correctly, it is currently only possible to set up a fresh cluster in KRaft mode, but not to migrate an existing one with ZooKeeper. Kafka 3.5 will be a migration version they intend you to migrate from ZooKeeper to KRaft.
This is explained quite nicely in the release notes from Kafka, especially for Kafka 3.3 and the release video
As long as your Kafka brokers are not running with Java 8 still, you can simply do a rolling upgrade from 2.7 to 3.X like you are used to.
We have installed zookeeper 3.6.2 and Kafka 2.13-2.6.0
Recently I noticed that zookeeper-server-start.sh is actually using the zookeeper jar file from /kafka/libs/ which is zookeeper-3.5.8.jar.
How do I upgrade zookeeper to 3.6.2. Do I have to find a version of kafka that has it bundled in the tar?
Why do they ask you to download and install zookeeper if the jar is already bundled in kafka and that is the one that kafka uses?
You will need to separately install (and start) a Zookeeper cluster outside of the versions and scripts provided by Kafka if you want to upgrade it, but there are risks involved.
Kafka does come bundled with Zookeeper and is tested against specific versions, but it is not upgraded as frequently as Zookeeper project does releases. You should follow the Kafka upgrade notes before blindly upgrading dependency components; there are some notes around version 2.6 that mentions situations where Zookeeper upgrades can fail and what settings need to be added to fix it. And unless you can guarantee a Zookeeper server upgrade will be backwards compatible, Kafka still has a version-specific Zookeeper client for network communications.
Worth pointing out that Kafka KRaft (Zookeeper-less mode) is already in preview release phase with newest 3.x releases, and is scheduled to be closer to production-ready by the end of the year, so I would hold off on upgrading Zookeeper unless you actually need if for something else.
I have 5 kafka clusters and one of them runs kafkamirror to consume from the other 4 kakfa and produce for the main kafka cluster.
I've tried the latest version of kafka separately and my apps seems to work fine with the newest version. The upgrade process, as described in the Kafka upgrade https://kafka.apache.org/documentation/#upgrade is a rolling update, first going to 2.5.0 and maintaining the broker protocol with 0.10.0 and then a rolling restart with the newer protocol
The problem is the mirrormaker and have no downtime in the upgrade process to avoid loss of messages. I've read that consumers should be upgraded first, in this case mirrormaker is producer and consumer at the same time, so I tryed to upgrade my main kafka cluster first and now is running with 2.5.0 with 2.5.0 protocol and the other cluster of my tests it's still on 0.10.0. When I try to initiate the mirror I am getting some warnings/errors because I am still using old deprecated configs:
Which path should I follow to do the upgrade?
i have following problem
I am using java 8 version for this and zookeeper is not working for this.
Downloads$ cd confluent-5.2.2/
roshni#roshni-HP-Pavilion-15-Notebook-PC:~/Downloads/confluent-5.2.2$
/home/roshni/Downloads/confluent-5.2.2/bin/confluent start
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
WARNING: Java version 1.8 or 1.11 is recommended.
See https://docs.confluent.io/current/installation/versions-interoperability.html
What you did is correct. Based on the warning, it seems to suggest you're not using Java 8 or 11, though, so check your JAVA_HOME variable, for example.
You could try running confluent logs to see if you get more information, and just do confluent start kafka if you're truly only trying to run Kafka
Otherwise, you could explicitly start Zookeeper and Kafka manually, as well as other Confluent services, using the other scripts in the bin folder, just the same as any other Kafka install
I need to use confluent schema-registry to connect to 0.9 kafka. But for schema-registry 2.0.1 version I didn't see SSL configuration available. Is there a way to enable ssl for schema-registry and kafka-rest to talk to 0.9 kafka?
That functionality was only added as of the 3.0 release line. If you want support for 0.9/2.0 releases, you could try either a) taking a 3.0 release and reducing the Kafka dependency to 0.9 versions (this might work, but I think there were other unrelated changes that might require additional changes), or b) backport the SSL changes to a 2.0 version (although I can't be sure off the top of my head whether this version supports everything required to enable security support).
Note that if you follow the upgrade docs, upgrading brokers to enable use of newer client versions is generally relatively painless these days.