Is there any way to connect Kafka as sink in Google Dataflow? I know we can use CloudPubSubConnector with pub/sub and Kafka, but I dont want to use Pub/sub in between Dataflow and Kafka.
Thanks,
Bala
Yes (assuming you are using Java SDK). See 'Writing to Kafka' with usabe example in JavaDoc for KafkaIO : https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L221
If you're writing DataFlow jobs in Python you can use Confluents Kafka client
[https://github.com/confluentinc/confluent-kafka-python][1]
and write you own Beam Sink/Source interface. There is a guide for writing your own sinks/sources in Beam [https://beam.apache.org/documentation/sdks/python-custom-io/][1]
Related
I was going through the https://learn.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-producer-consumer-api looks like there is no way if we can use the Spring Cloud Stream API with the Azure Kafka. Is this understanding?
If I've to implement the Kafka Producer and Consumer still I've to use the Kafka Client API only?
I am also looking to push and read messages into Azure Kafka by using the Spring batch Approach. Can I still use KafkaItemReader and KafkaItemWriter API provided by Spring Batch with the Azure Kafka? Does it works well?
Yes, you can use the Spring libraries (Stream and Batch); they use the same kafka-clients APIs.
I want to live stream from one system to another system .
I am using kafka-python and am able to live stream locally.
Figures out that connectors will handle multiple devices. Can someone suggest me a way to use connectors to implement it in python?
Kafka Connect is a Java Framework, not Python.
Kafka Connect runs a REST API which you can use urllib3 or requests to interact with it, not kafka-python
https://kafka.apache.org/documentation/#connect
Once you create a connector, you are welcome to use kafka-python to produce data, which the JDBC sink would consume, for example, or you can use pandas for example to write to a database, which the JDBC source (or Debezium) would consume
I'm currently planning the architecture for an application that reads from a Kafka topic and after some conversion puts data to RabbitMq.
I'm kind new for Kafka Streams and they look a good choice for my task. But the problem is that Kafka server is hosted at another vendor's place, so I can't even install Cafka Connector to RabbitMq Sink plugin.
Is it possible to write Kafka steam application that doesn't have any Sink points, but just processes input stream? I can just push to RabbitMQ in foreach operations, but I'm not sure will Stream even work without a sink point.
foreach is a Sink action, so to answer your question directly, no.
However, Kafka Streams should be limited to only Kafka Communication.
Kafka Connect can be installed and ran anywhere, if that is what you wanted to use... You can also use other Apache tools like Camel, Spark, NiFi, Flink, etc to write to RabbitMQ after consuming from Kafka, or write any application in a language of your choice. For example, the Spring Integration or Cloud Streams frameworks allows a single contract between many communication channels
I have a requirement in my project where I have to get the data after hitting a REST service. Is there any Kafka connector that does this work or I have to write the custom code using streams or producer ?
I tried finding REST connector on ( https://www.confluent.io/hub/ ) but could not find anything. Can you please suggest?
Seems you're asking for the HTTP Sink Connector, which does exist on that page.
There is also a kafka-connect-rest project on Github
Since IIB v10.0.0.7 I can use KafkaConsumer node to receive messages that was published on a Kafka topic.
I need some client which will be able to recieve message from Kafka and put it in IBM MQ and get message from IBM MQ and publish in to Kafka topic. I alredy have IIB and IBM MQ. Kafka is messaging system of one of integration systems.
Can I somehow put received message from Kafka in IBM queue using KafkaConsumer node and MQOutput node ? Or get message from queue with MQOutput and publish it to Kafka topic with KafkaProducer node ?
Or it is not a good idea to mix this tecnologies in such a way and should look for some another workaround ?
Hi you could use Kafka Connect connectors.
https://www.confluent.io/product/connectors/
There are community connectors for MQ.
Alternatively, if you're using IBM MessageHub, i.e. Kafka-as-a-service in the IBM Cloud, there you can have an MQ-to-Kafka bridge ran as a service itself.
https://console.bluemix.net/docs/services/MessageHub/messagehub088.html#bridges
I hear this question every week...
The article “Apache Kafka vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? (https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/)” discusses why Kafka is not competitive but complementary to integration and messaging solutions (including IBM MQ) and how to integrate both.
IIB can write to IBM MQ and one could IBM MQ source connector to write to kafka.
https://docs.confluent.io/kafka-connect-ibmmq-source/current/
Usage of kafka or IIB will be use to use dependent. Kafka is your messaging platform with persistence, ability connect to different sources and sinks and if needed enrich the messages on the fly in real/near realtime.