Since IIB v10.0.0.7 I can use KafkaConsumer node to receive messages that was published on a Kafka topic.
I need some client which will be able to recieve message from Kafka and put it in IBM MQ and get message from IBM MQ and publish in to Kafka topic. I alredy have IIB and IBM MQ. Kafka is messaging system of one of integration systems.
Can I somehow put received message from Kafka in IBM queue using KafkaConsumer node and MQOutput node ? Or get message from queue with MQOutput and publish it to Kafka topic with KafkaProducer node ?
Or it is not a good idea to mix this tecnologies in such a way and should look for some another workaround ?
Hi you could use Kafka Connect connectors.
https://www.confluent.io/product/connectors/
There are community connectors for MQ.
Alternatively, if you're using IBM MessageHub, i.e. Kafka-as-a-service in the IBM Cloud, there you can have an MQ-to-Kafka bridge ran as a service itself.
https://console.bluemix.net/docs/services/MessageHub/messagehub088.html#bridges
I hear this question every week...
The article “Apache Kafka vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? (https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/)” discusses why Kafka is not competitive but complementary to integration and messaging solutions (including IBM MQ) and how to integrate both.
IIB can write to IBM MQ and one could IBM MQ source connector to write to kafka.
https://docs.confluent.io/kafka-connect-ibmmq-source/current/
Usage of kafka or IIB will be use to use dependent. Kafka is your messaging platform with persistence, ability connect to different sources and sinks and if needed enrich the messages on the fly in real/near realtime.
Related
We are currently using IBM MQ for messaging and moving to ActiveMQ, but the challenge is there are few producers who would continue to use IBM MQ for sometime whereas consumers are ready to migrate.
Is there a way we can bridge IBM MQ and ActiveMQ Artemis, so that any message that arrives in IBM MQ queue should get auto replicated in ActiveMQ and consumer pick up from there ( same goes in reverse order)?
Implementing a new service which consume from IBM MQ and put of ActiveMQ does not seem feasible as there are huge list of services and this redundant work does not seems feasible at enterprise level.
I am new to Spring Cloud Data Flow, and need to listen for messages on a topic from an external kafka cluster. This external kafka topic in confluent cloud would be my Source that I need to pass on to my Sink application.
I am also using kafka as my underlying message broker, which is a separate kafka instance that is deployed on kubernetes. I'm just not sure what is the best approach to connect to this external kafka instance. Is there an existing kafka Source app that I can use, or do I need to create my own Source application to connect to it? Or is it just some kind of configuration that I need to setup to get connected?
Any examples would be helpful. Thanks in advance!
'I am working on a Kafka Integration, where I a publishing and reading messages from topic.
I am able to publish messages to topic but unable to read those messages using inbound endpoint.
When I checked WSO2 documentation, they are recommending Kafka kafka_2.9.2-0.8.1.1 which is very old.
Can't we use the latest version of Kafka with Micro Integrator? Can any one suggest please.'
If you are using the Kafka Inbound endpoint from the WSO2 connector store, then you should be able to use the latest versions. Checkout this documentation https://github.com/wso2-extensions/esb-inbound-kafka/blob/release-1.0.7/docs/config.md
Also checkout this github issue thread https://github.com/wso2/product-ei/issues/2239
I have a scenario where we are forwarding our application logs to Kafka topic using fluentD agents,
as Kafka team introduced Kerberos authentication and fluentD version not supporting this authentication, I cannot directly use forward logs.
Now we have introduced a new Kafka server without authentication and created a topic there, I want forward messages from this topic in the new server to another topic in another server using Kafka connectors,
want to know how I can achieve this?
There's several different tools that enable you to stream messages from a Kafka topic on one cluster to a different cluster, including:
MirrorMaker (open source, part of Apache Kafka)
Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
Disclaimer: I work for Confluent.
High level Design of application :
Upstream system sends stream of data, data is received by Java Application. Using KAFKA as data store, logstash will publish stored data in Elastic index, and all the application will use elastic search query to get the data.
Problem : To publish data from Java application to KAFKA, which API Kafka JMS client or Java Kafka Producer/Consumer API should be used?
As per kafka documentation, If you are interested in writing new Java applications then you are encouraged to use the Java Kafka Producer/Consumer APIs as they provide advanced features not available when using the kafka-jms-client https://docs.confluent.io/current/clients/kafka-jms-client/docs/index.html .
Also as per KAFKA documentation it is not typical Messgaing broker and not all JMS concepts map 1:1 kafka.
Is there any benefit of using JMS API for KAFKA since KAFKA is not typical Messaging broker [and application will be still tightly couple to KAFKA] and not all JMS concepts can be mapped to kafka?