How to consume Kafka Topic in Consumer MQ Topic - apache-kafka

Question

I have a requirement where I need to consume Kafka Topic and write it into MQ Topic. Can someone advise me the best way to do it, I am new to Kafka.
I have read about the IBM MQ Connector in confluent but could not get the idea how to implement it.

Answer 1

The best way to move data from Kafka to MQ is to use the IBM MQ sink connector: https://github.com/ibm-messaging/kafka-connect-mq-sink
This is a Kafka Connect connector. The README contains details for building and running it.

Answer 2

Kafka has a component called Kafka Connect. It is used to read and write data to/from Kafka into other systems such as Database in your case MQ.
Kafka connect can have two kind of connectors
Source connectors - Read data from an external system and write to Kafka (For eg. Read inserted/modified rows from a table in DB and insert into a topic in Kafka)
Sink Connector - Read message from Kafka write to external system.
The link you have added is a source connector, it will read messages from the MQ and write to Kafka.
For simple use case you do not need Kafka connect. You can write a simple Kafka consumer that will read data from Kafka topic and write it to MQ.
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
//Insert code to append to MQ here
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}