I've got two brokers. the first runs 0.9 and the second runs 0.10
Various workers and daemons consume and produce messages on both brokers.
For one application, I need messages from a 0.9 topic consumable from an application that's using KStreams and is connected to the 0.10 broker.
Is there a straightforward way of copying just the one topic from 0.9 to 0.10? Or using the 0.10 clients to connect to 0.9? I'd hate to have to resort to cramming both versions in the same jar. Just consuming the 0.9 broker with a 0.10 client doesn't seem to work.
In general, only Kafka broker are backward compatible (not Kafka client). Thus, a client can connect to newer brokers, but not to older once.
Because Kafka Streams library uses 0.10.x client, it only works with 0.10.x brokers.
Thus, upgrading your broker as described here: https://kafka.apache.org/documentation.html#upgrade should be the best way to go (this is safe, as brokers are backward compatible, thus not breaking any other applications using this broker with older clients).
As an alternative, you could also use Mirror Maker to replicate the topic from 0.9.x cluster to 0.10.x cluster.
my solution was to use jarjar to rewrite the 0.9 clients jar so that the classes/types don't conflict with the 0.10 client. it's dirty, but it works around the jvm's opinion on having two versions of the same library.
Related
The question is simple, what is the difference between those two guarantees in Kafka Streams?
processing.guarantee: exactly_once / exactly_once_beta
Docs says
Using "exactly_once" requires broker version 0.11.0 or newer, while using "exactly_once_beta" requires broker version 2.5 or newer. Note that if exactly-once processing is enabled, the default for parameter commit.interval.ms changes to 100ms.
But there's nothing about difference.
When you configure exactly_once_beta, transaction processing will be done using a new implementation, enabling better performance as the number of producers increases.
Note however that a two-step migration will be necessary if you have been using exactly_once with an earlier Kafka version.
Currently we are using Apache kafka_2.11-0.9.0.1 and Apache Solr 5.5 with Zookeeper 3.4.6.
But we are upgrading Apache Solr, hence need to upgrade Zookeeper to 3.4.12.
Kafka is working with this zookeeper version as per our basic testing done. But we just want to confirm whether or not Zookeeper 3.4.12 is officially supported with kafka_2.11-0.9.0.1
Yes, it will work (just tested), but without backing up the Kafka data in Zookeeper and restoring it to the new one, then you will lose all the Kafka data, meaning your topics and committed offsets will be lost.
FWIW, it might be worth upgrading Kafka as well.
The old version suggests that consumer connect zookeeper, and the new version suggests linking broker? A netizen from a community replied to me that the old version of topic's offset is ZK, and the new version is Kafka itself. Is this the answer to this answer?
Older versions of Kafka i.e before 0.9 version store offsets with Zookeeper.
Newer version of Kafka, store offsets in an internal Kafka topic called __consumer_offsets.
The newer version still provides the feasibility to store offsets with Zookeeeper.
With this the consumers can now only talk to the Brokers and does not need to rely on Zookeeper.
If there are many consumers simultaneously reading from Kafka, the read write load on ZooKeeper may exceed its capacity, making ZooKeeper a bottleneck.
check this for more information
https://github.com/SOHU-Co/kafka-node/issues/502
Is that possible?
The official .Net client confluent-kafka-dotnet only seems to provide consumer and producer functionality.
And (from what I remember looking into Kafka streams quite a while back) I believe Kafka Streams processors always run on the JVMs that run Kafka itself. In that case, it would be principally impossible.
Yes, it is possible to re-implement Apache Kafka's Streams client library (a Java library) in .NET. But at the moment there doesn't exist such a ready-to-use Kafka Streams implementation for .NET.
And (from what I remember looking into Kafka streams quite a while back) I believe Kafka Streams processors always run on the JVMs that run Kafka itself. In that case, it would be principally impossible.
No, Kafka Streams "processors" as you call them do not run in (the JVMs of) the Kafka brokers, which would be server-side.
Instead, the Kafka Streams client library is used to implement client-side Java/Scala/Clojure/... applications for stream processing. These applications talk to the Kafka brokers (which form the Kafka cluster) over the network.
May 2020 there seems to be a project in the making to support Kafka Streams in .NET:
https://github.com/LGouellec/kafka-stream-net
As per their road-map they are now in early beta and intend to get to v1 but the end of the year or beginning of next
I am using spring-cloud-stream-kafka. I am starting 2 instances of a consumer application with the same group name testGroup consuming messages from a Kafka topic testTopic with single partition, I expected one of the consumer instances to consume messages from the Kafka topic and the other instance to not consume anything since they share the same group name. I am using 0.8.xx version of Kafka client. Here are my properties
spring.cloud.stream.bindings.input.destination=testTopic
spring.cloud.stream.bindings.input.group=testGroup
Any idea what could be happening ?
Right, that is the expected behaviour for 1.0.3/ 0.8 where we use the SimpleConsumer API. You don't need to do that after 1.1 - see http://docs.spring.io/spring-cloud-stream/docs/1.0.3.RELEASE/reference/htmlsingle/index.html#_instance_index_and_instance_count for details.
Which version of Spring Cloud Stream are you using?
I think you are using 1.0.3 version of Spring Cloud Stream which supports Kafka 0.8.xx version. Anyway, this looks like a bug.
Also, I don't see this issue when using Kafka 0.9 or 0.10 with Spring Cloud Stream 1.1.x.