'I am working on a Kafka Integration, where I a publishing and reading messages from topic.
I am able to publish messages to topic but unable to read those messages using inbound endpoint.
When I checked WSO2 documentation, they are recommending Kafka kafka_2.9.2-0.8.1.1 which is very old.
Can't we use the latest version of Kafka with Micro Integrator? Can any one suggest please.'
If you are using the Kafka Inbound endpoint from the WSO2 connector store, then you should be able to use the latest versions. Checkout this documentation https://github.com/wso2-extensions/esb-inbound-kafka/blob/release-1.0.7/docs/config.md
Also checkout this github issue thread https://github.com/wso2/product-ei/issues/2239
Related
I have a scenario where we are forwarding our application logs to Kafka topic using fluentD agents,
as Kafka team introduced Kerberos authentication and fluentD version not supporting this authentication, I cannot directly use forward logs.
Now we have introduced a new Kafka server without authentication and created a topic there, I want forward messages from this topic in the new server to another topic in another server using Kafka connectors,
want to know how I can achieve this?
There's several different tools that enable you to stream messages from a Kafka topic on one cluster to a different cluster, including:
MirrorMaker (open source, part of Apache Kafka)
Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
Disclaimer: I work for Confluent.
Is it possible to connect cometD client with Kafka producer? Any suggestions?
Currently I am having a CometD client in python which is extracting data real time from a Salesforce object.
Now I want to push that data into Kafka producer. Is it possible to do that? And how?
Solved.
By using https://github.com/dkmadigan/python-bayeux-client to extract the events from Salesforce, I was able to push into the Kafka broker.
I am trying to find the api that connects kinesis to kafka.
The API listed on the kafka connecter website is dead.
see https://github.com/jcustenborder/kafka-connect-kinesis
Any suggested replacement?
This connector will be included in the Confluent Enterprise connector program, and will be available for download soon from https://confluent.io
Since IIB v10.0.0.7 I can use KafkaConsumer node to receive messages that was published on a Kafka topic.
I need some client which will be able to recieve message from Kafka and put it in IBM MQ and get message from IBM MQ and publish in to Kafka topic. I alredy have IIB and IBM MQ. Kafka is messaging system of one of integration systems.
Can I somehow put received message from Kafka in IBM queue using KafkaConsumer node and MQOutput node ? Or get message from queue with MQOutput and publish it to Kafka topic with KafkaProducer node ?
Or it is not a good idea to mix this tecnologies in such a way and should look for some another workaround ?
Hi you could use Kafka Connect connectors.
https://www.confluent.io/product/connectors/
There are community connectors for MQ.
Alternatively, if you're using IBM MessageHub, i.e. Kafka-as-a-service in the IBM Cloud, there you can have an MQ-to-Kafka bridge ran as a service itself.
https://console.bluemix.net/docs/services/MessageHub/messagehub088.html#bridges
I hear this question every week...
The article “Apache Kafka vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? (https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/)” discusses why Kafka is not competitive but complementary to integration and messaging solutions (including IBM MQ) and how to integrate both.
IIB can write to IBM MQ and one could IBM MQ source connector to write to kafka.
https://docs.confluent.io/kafka-connect-ibmmq-source/current/
Usage of kafka or IIB will be use to use dependent. Kafka is your messaging platform with persistence, ability connect to different sources and sinks and if needed enrich the messages on the fly in real/near realtime.
I have a requirement where messages are coming in json format from ACTIVEMQ and I have to expose end point where I can receive messages. I am using KAFKA but don't know whether can I use KAFKA REST API to receive json messages and store them against a particular topic in Kafka broker?
There is no REST API in the core Apache Kafka distribution but there is a very good open source REST Proxy for Kafka available from Confluent. It enables both publish and subscribe via REST APIs. The code and docs are on github at https://github.com/confluentinc/kafka-rest or you can download the entire Confluent open source platform which includes Apache Kafka, REST Proxy, Schema Registry and a few other useful enhancements at http://www.confluent.io/download/