The documentation on spring.io only have kafka and rabbitMQ,
The demo on github(spring-cloud-alibaba-example) only have old version,
How to config rocketMQ in spring-cloud-stream-3.2.2 or spring-cloud-alibaba-2021.0.1.0?
Related
env
apache kafka 2.7
apache flume 1.9.0
trouble
I'm gonna setup kafka cluster with version 2.7.
Using DNS attribute(use_all_dns_ips), I want to perform traffic switching by dividing the dns of both clusters in case of DR.
But now, i using Apache Flume tailing source. I cannot apply DNS attribute values in flume because flume channel doesn't support DNS properties.
I also looked for the kafka source connector, but in the official version, there seems to be only a spooldir version and no tailing source version.
Is there any tailing source application that you can recommend to add a dns attribute?
Currently we are using Apache kafka_2.11-0.9.0.1 and Apache Solr 5.5 with Zookeeper 3.4.6.
But we are upgrading Apache Solr, hence need to upgrade Zookeeper to 3.4.12.
Kafka is working with this zookeeper version as per our basic testing done. But we just want to confirm whether or not Zookeeper 3.4.12 is officially supported with kafka_2.11-0.9.0.1
Yes, it will work (just tested), but without backing up the Kafka data in Zookeeper and restoring it to the new one, then you will lose all the Kafka data, meaning your topics and committed offsets will be lost.
FWIW, it might be worth upgrading Kafka as well.
We are migrating our application from Apache Kafka to Confluent Platform .
Apache Kafka version:1.1.0
Confluent :4.1.0
Tried these options:
Manually copying the zookeeper logs and Kafka Logs- Not an optimal way
because of volume and data correctness.
Mirror Maker - This will replicate newly created topics and ACL. It will not
migrate old details in Apache Kafka
Please suggest better approaches on this.
You can keep your existing Kafka and Zookeeper installation.
Confluent does not change any way these run or manage data.
You can configure the REST Proxy, Schema Registry, Control Center, KSQL, etc. to use your existing bootstrap servers or Zookeeper connection; nothing should need migrated, you're only adding extra consumer/producer services which just happen to be provided by Confluent.
If you later plan on upgrading your brokers, then you can start up new ones from the Confluent package, migrate the partitions, then shut down the old ones. Similarly for Zookeeper, but make sure that you have at least 2 up during this process, and always have an odd number of them available after your transition
I am trying to upgrade from the apache kafka to the confluent kafka
As the storage of the temp folder is quite limited I have changed the log.dirs of server.properties to a custom folder
log.dirs=<custom location>
Then try to start kafka server via the Confluent CLI (version 4.0) using below command :
bin/confluent start kafka
However when I check the kafka data folder, the data still persitted under the temp folder instead of the customzied one.
I have tried to start kafka server directly which is not using the Confluent CLI
bin/kafka-server-start etc/kafka/server.properties
then seen the config has been picked up properly
is this a bug with confluent CLI or it is supposed to be
I am trying to upgrade from the apache kafka to the confluent kafka
There is no such thing as "confluent kafka".
You can refer to the Apache or Confluent Upgrade documentation steps for switching Kafka versions, but at the end of the day, both are Apache Kafka.
On a related note: You don't need Kafka from the Confluent site to run other parts of the Confluent Platform.
The confluent command, though, will read it's own embedded config files for running on localhost only, and is not intended to integrate with external brokers / zookeepers.
Therefore, kafka-server-start is the production way to run Apache Kafka
Confluent CLI is meant to be used during development with Confluent Platform. Therefore, it currently gathers all the data and logs under a common location in order for a developer to be able to easily inspect (with confluent log or manually) and delete (with confluent destroy or manually) such data.
You are able to change this common location by setting
export CONFLUENT_CURRENT=<top-level-logs-and-data-directory>
and get which location is used any time with:
confluent current
The rest of the properties are used as set in the various .properties files for each service.
With Kafka Nodejs client (nodejs is kafka producer), we could find kafka-node with zookeeper integration. Is there similar package for Java as Kafka producer with integration of zookeeper. The purpose is that Java could talk to zookeeper only and let zookeeper to find kafka brokers so that even one broker is down, zookeeper would advice producer client to talk with the newly selected primary broker.