We are planning to upgrade Kafka broker to 2.12.X but our Kafka clients are still going to use 0.10.x or higher versions
On local, we have verified and not seen any issues in producing and consuming with older client versions mentioned above while the broker is upgraded to kafka_2.12-2.3.0
Is there a compatibility matrix for Kafka broker and client versions mentioned? Did anyone face any issues with such upgrades?
PS -
I went through below link
https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
As the link says, broker 1.0.0 (and up) will work for basic client interaction on any client that supports features added in KIP-35
The main missing features of clients before 0.11 or 1.0 would be the message headers and idempotent producer or exactly-once processing semantics for Kafka Streams
You'll also want to be careful on upgrading the log format version too soon, because as the Kafka upgrade steps say, you should only change that once most of the clients have upgraded
Related
I'm running a Kafka Streams application. Here's the current version compatibility:
kafka-streams v1.1.0 (with custom changes on top of it)
kafka cluster v2.1.1
I'm planning to upgrade my Kafka cluster to v2.8.1. Is the same kafka-streams library compatible with the newer kafka version? I'm not able to find any official compatibility matrix for streams. I could find one from Confluent, but none from the official Kafka website.
I ran my Kafka Streams app against Kafka v2.8.1 and it does seems to run. I'd like to avoid any surprises at a later point in time, hence looking for any pointers to compatibility or support.
The Kafka Streams API uses the Kafka consumer and producer API underneath. The Kafka protocol is backward compatible so you should not have any problem (so having new Kafka cluster working with old Kafka clients).
Takes only into account that starting with Kafka 4.0.0 (not released yet), they are planning to remove this backward compatibility. See KIP-896 for more details here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
Trying to understand compatibility of MM2 libraries packaged as part of core kafka release 2.4 with Kafka Cluster running with Kafka core version 2.0.0.
The major improvements to MM2 got implemented as part of the KIP-382 got merged via this PR.
In this PR I don’t see any changes being made to core kafka. The changes are only seen with Kafka-connect and mirrormaker related code.
As I know Kafka Connect is a client application to kafka (Similar to any standard consumer and producer application). And MM2 built on top of Kafka Connect Architecture.
Though MirrorMaker2 related libraries are bundled with Core kafka libraries i.e., 2.4 Kafka version as part of release cycles and packages, I think MM2 doesn't have absolute dependency on the same version of the Kafka Broker (i.e., the one built with kafka 2.4.).
The reason being MM2 component is client side component.
This is valid to assume that we can set MM2 using kafka core 2.4 release packages to work with Kafka Broker set up with Kafka 2.0.0?
Any responses, comments or any materials in this regard are highly appreciated. Thank you.
we are using Apache kafka 2.0 in our production environment and now we are planning to upgrade the kafka version to 2.6 from 2.0
we are running in three broker based cluster setup
i am having the below questions.
1)is it possible to upgrade the kafka from one version to higher version?
2)while upgrading is there any data loss happen?
3)is it possible to perform while the cluster is running?
4)How to rollback to the down version if something wrong happened?
can you share your valuable thoughts for this question..
it would be helpful to setup..
Yes, upgrades are possible - http://kafka.apache.org/26/documentation.html#upgrade
Data that's already written to the topics shouldn't get lost if you follow the guide. Active clients might experience network exceptions, retries, and potential dropped packets while individual brokers are restarting.
A rolling upgrade is possible to prevent downtime
Depending on the exact version, rollbacks are not possible due to internal log format changes (as indicated in the documentation)
I am using kafka client & streams library version 2.7.0 for building my application. However the kafka brokers(2 different clusters) are at older version ( 2.4.1 & 2.6.0).
As i understand we can use the latest clients & Streams library and it should run fine with older version of kafka brokers. Am i correct ? Is there any compatibility matrix between client & streams library with kafka brokers ?
I tried running in my application (with 2.7.0 client library) in local environment ( with kafka version 2.6.0) and it worked fine but wanted to get the supported compatibility between them
Update: As onecricketeer has helpfully pointed out, you can refer to the Kafka Compatability Matrix. He also notes:
There is a general answer. Clients above 0.10.2 work with brokers down to that version for all basic functionality until stated otherwise. Extra functionality includes transactional/idempotence and record headers, which Spring may depend on, but Kafka Streams natively has no dependency on.
Additionally, the upgrade section of the Kafka Documentation provides guidance on upgrade order for various Kafka versions.
The compatability matrix provided by the spring-cloud-stream project may also be of assistance.
Should we update our Scala Kafka client library dependency (currently 0.10.2) to match the Kafka version of the broker (v1.1.0) ?
The Kafka 0.10.2 Documentation mentions
Starting with version 0.10.2, Java clients (producer and consumer)
have acquired the ability to communicate with older brokers. Version
0.10.2 clients can talk to version 0.10.0 or newer brokers
Are there any adverse effects when the client API version lags behind the server version? More importantly, can we safely update our Kafka client API library from 0.10.2 to 1.10?
While the brokers are now compatible with older clients, there are a few drawbacks in using older clients.
The main one is Message conversion. Between 1.1 and 0.10.2, the record format has changed. So, by default, older clients will force brokers to convert messages when producing and consuming. Conversion is pretty memory intensive and has a performance cost. See http://kafka.apache.org/documentation/#upgrade_11_message_format
Then obviously old clients are unable to use new features. Between 0.10.2 and 1.1, there's a ton a nice features like Exactly Once semantics, better authentication feedback on failure, Admin operations, etc