I need to use confluent schema-registry to connect to 0.9 kafka. But for schema-registry 2.0.1 version I didn't see SSL configuration available. Is there a way to enable ssl for schema-registry and kafka-rest to talk to 0.9 kafka?
That functionality was only added as of the 3.0 release line. If you want support for 0.9/2.0 releases, you could try either a) taking a 3.0 release and reducing the Kafka dependency to 0.9 versions (this might work, but I think there were other unrelated changes that might require additional changes), or b) backport the SSL changes to a 2.0 version (although I can't be sure off the top of my head whether this version supports everything required to enable security support).
Note that if you follow the upgrade docs, upgrading brokers to enable use of newer client versions is generally relatively painless these days.
Related
we have production Kafka cluster with 2.7 version , 5 Kafka brokers on RHEL 7.9 version
we want to upgrade the Kafka version to 3.X version
3.X version not include zookeeper , so we are wondering if we can do upgrade without any data loss
regarding to kafka 2.7 version , Kafka storing the metadata on zookeeper servers ( as brokers ids , topics names etc )
but is it possible to perform rolling upgrade from 2.7 to 3.x version without any data loss?
The upgrade guide should contain all infos you need.
While KRaft mode (without ZooKeeper) is production ready since 3.3, they still keep ZooKeeper around for compatibility until the 4.0 release.
Furthermore If I understand correctly, it is currently only possible to set up a fresh cluster in KRaft mode, but not to migrate an existing one with ZooKeeper. Kafka 3.5 will be a migration version they intend you to migrate from ZooKeeper to KRaft.
This is explained quite nicely in the release notes from Kafka, especially for Kafka 3.3 and the release video
As long as your Kafka brokers are not running with Java 8 still, you can simply do a rolling upgrade from 2.7 to 3.X like you are used to.
What kind of benefits do you think upgrading the kafka version will provide and is it important? What are the advantages of switching from 2.8 to 3.1?
The Kafka release notes cover all added features and fixes. For example, as of Kafka 3.3.1 release, Zookeeper is no longer required.
Kafka 3.0 removed support for Scala 2.11, and maybe Java 7/8, and enables transactions by default, I believe.
Make sure that you read the upgrade notes for protocol changes that will make it impossible to downgrade in the event of errors.
We have installed zookeeper 3.6.2 and Kafka 2.13-2.6.0
Recently I noticed that zookeeper-server-start.sh is actually using the zookeeper jar file from /kafka/libs/ which is zookeeper-3.5.8.jar.
How do I upgrade zookeeper to 3.6.2. Do I have to find a version of kafka that has it bundled in the tar?
Why do they ask you to download and install zookeeper if the jar is already bundled in kafka and that is the one that kafka uses?
You will need to separately install (and start) a Zookeeper cluster outside of the versions and scripts provided by Kafka if you want to upgrade it, but there are risks involved.
Kafka does come bundled with Zookeeper and is tested against specific versions, but it is not upgraded as frequently as Zookeeper project does releases. You should follow the Kafka upgrade notes before blindly upgrading dependency components; there are some notes around version 2.6 that mentions situations where Zookeeper upgrades can fail and what settings need to be added to fix it. And unless you can guarantee a Zookeeper server upgrade will be backwards compatible, Kafka still has a version-specific Zookeeper client for network communications.
Worth pointing out that Kafka KRaft (Zookeeper-less mode) is already in preview release phase with newest 3.x releases, and is scheduled to be closer to production-ready by the end of the year, so I would hold off on upgrading Zookeeper unless you actually need if for something else.
I have seen that i can either
(1) Use the ZooKeeper that comes with Kafka, or
(2) Use the ZooKeeper from Apache itself.
Which is the preferred method(if there is one) and why? My use-case is for a small application, so it will be a 3 ZooKeeper ensemble/cluster. I am using Window 10 for my test. The ZooKeeper version i am using is 3.5.6. The Apache Kafka version i am using is 2.12-2.3.0
Note:
I have tried both ways i.e (1) and (2), and both work.
UPDATE:
Found what i was looking for. For use case (2), if i want to use Kafka 3.0.0, ideally, i will use it with Zookeeper 3.6.3 as that's what it has been tested with, as noted here.
ZooKeeper has been upgraded to version 3.6.3.
Kafka is tested against the Zookeeper version it comes with.
If you want to upgrade, you'll need to verify Zookeeper itself is backwards compatible with older clients/protocols that Kafka may use.
Its unlikely that jumping from 3.4.x to 3.5.x is a compatible change, but if you stay within the same minor release, it should be fine
Is the kafka consumer 0.10 compatible with 0.9 broker?
If I'm not mistaken the 0.9 consumer is still considered beta whereas 0.10 is stable, right? That's why I'm interested in using the 0.10 version but my broker version is 0.9 and I wouldn't like to upgrade that yet.
If you want to use 0.10 clients you need to upgrade your cluster to 0.10.
Kafka is backward compatible with regards to clients but not forward compatible. That is, a 0.9 client can use a 0.10 cluster but a 0.10 client can not use a 0.9 cluster.
The idea is to upgrade your cluster first to 0.10, and then gradually migrating clients from 0.9 to 0.10.
My answer is not only for 0.10.0 client. I search Kafka client & broker version compatibility and reach here, so I leave a more complete answer for future reader.
According to this official post:
The “bidirectional” client compatibility work done in KIP-35 and KIP-97 removed these limitations. New Java clients can now communicate with old brokers.
Improved client compatibility is a new feature in Kafka 0.10.2. It is supported by brokers that are at version 0.10.0 or later.
For example, if we use client in 2.0.0, we can use broker 0.10.0 and all other later version (and of course, new feature will not supported). But if we use client in 0.10.1, we can only communicate with broker from 0.10.1 and later version.
So, kafka consumer 0.10 is not compatible with 0.9 broker.
You didn't say what language your client is written in and which client libraries you are using. Some clients (like those based on librdkafka) can handle connections to an older broker but the general rule (which is also true for the default Apache kafka java clients) is that the broker must be of equal or higher version number than the clients. In other words, Kafka is backward compatible, but it is not yet fully forward compatible.