Has anyone in Fineract-CN replaced ActiveMQ with Kafka?
I want to use Kafka in Fineract-CN modules, but the Spring Boot version is currently 1.4.1 and is not possible.
The Fineract-CN documentation says:
Apache ActiveMQ is used by default, but can be replaced with any Java Messaging Service (JMS) compatible message queue."
However, to my knowledge Kafka does not implement JMS. Therefore, it appears you won't be able to use Fineract-CN with Kafka.
Related
I'm running a Kafka Streams application. Here's the current version compatibility:
kafka-streams v1.1.0 (with custom changes on top of it)
kafka cluster v2.1.1
I'm planning to upgrade my Kafka cluster to v2.8.1. Is the same kafka-streams library compatible with the newer kafka version? I'm not able to find any official compatibility matrix for streams. I could find one from Confluent, but none from the official Kafka website.
I ran my Kafka Streams app against Kafka v2.8.1 and it does seems to run. I'd like to avoid any surprises at a later point in time, hence looking for any pointers to compatibility or support.
The Kafka Streams API uses the Kafka consumer and producer API underneath. The Kafka protocol is backward compatible so you should not have any problem (so having new Kafka cluster working with old Kafka clients).
Takes only into account that starting with Kafka 4.0.0 (not released yet), they are planning to remove this backward compatibility. See KIP-896 for more details here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
Is there any example (somewhere) of spring-remoting working on top of (spring-) Kafka ?
There is no Kafka implementation of spring-remoting.
spring-remoting itself is deprecated as of Spring Framework 5.3.
https://github.com/spring-projects/spring-framework/commit/d9ccd618ea9cbf339eb5639d24d5a5fabe8157b5
You can, however, use the RPC support in Spring for Apache Kafka
https://docs.spring.io/spring-kafka/docs/current/reference/html/#replying-template
While starting a topology with Kafka Spout with new Kafka version 2.1.0 and Storm version 1.2.2 getting a java.lang.ClassNotFoundException: kafka.api.OffsetRequest. I don't get this when I use Kafka version 0.10.0.1. Can you guys please help as I want to be on latest Kafka version?
I have tried all the latest kafka version starting with 2.*. But not of it works.Caused by: java.lang.ClassNotFoundException: kafka.api.OffsetRequest
kafka.api is the old Scala classes. Many of these were removed in Kafka 2.x
The majority of those classes were moved to org.apache.kafka.common.requests, and there is ListOffsetRequest and OffsetFetchRequest, so not sure which you're trying to use.
If Storm itself is depending on these old APIs, then you are bound to those, your own processor cannot use the new APIs.
Plus, the Kafka server version itself only supports certain API calls of these new request classes.
Adding to this answer, I suspect you're using the storm-kafka library for Kafka integration. You need to migrate to storm-kafka-client, which is based on the new Kafka APIs. The documentation for the new module can be found here.
If you need to migrate committed offsets from storm-kafka, you can use the utility at https://github.com/apache/storm/tree/master/external/storm-kafka-migration. It will let you migrate without having to start over on your Kafka partitions.
Any Opensource tool to monitor confluent Kafka? Most of the opensource tools available are specific to Apache Kafka but not for Confluent Kafka.
we want to monitor atleast the connectors, streams and cluster health
The Kafka that is distributed in the Confluent Platform is Apache Kafka. There really is no such thing as "Confluent Kafka". Any tools that work with the latest version of Apache Kafka (including Kafka Connect and Kafka Streams) will work with the same versions of Kafka included with Confluent Open Source.
Confluent 3.3 includes Apache Kafka 0.11
Confluent 3.2 includes Apache Kafka 0.10.2
Confluent 3.1 includes Apache Kafka 0.10.1
Confluent 3.0 includes Apache Kafka 0.10.0
Confluent 2.0 includes Apache Kafka 0.9
Confluent 1.0 includes Apache Kafka 0.8.2
Note: Confluent Enterprise includes its own monitoring and management GUI called Control Center. Control Center is a separate process so the Apache Kafka is still the same as the open source version.
You can use updated version of KafkaOffsetMonitor. It supports SSL/TLS and Kerbros. Also uses Kafka 1.1.0 library.
You should be able to use kafka-monitor for monitoring your cluster's health as well as Burrow and KafkaOffsetMonitor for monitoring your consumer application lag. Also, you should definitely use something like jmxtrans for collecting your Kafka broker metrics.
I have a topic in the WSO2 ESB 4.6.0 that I want to create subscriptions to at runtime using JMS, to allow pub-sub.
It is documented that the event-broker in 4.6.0 is based on Apache Qpid, however, it doesn't appear to be bundled. By default, an in-memory event broker is configured.
http://docs.wso2.org/wiki/display/ESB460/Eventing
I am using ActiveMQ as my JMS message broker, however, if required, it is possible to change to Qpid.
Has anyone managed to get this working?
Yes. You can use apache Qpid as your JMS broker with WSO2 ESB. It has been tested in production enviornments as well. You can follow the same steps as mentioned in the samples which uses ActiveMQ as the JMS broker.