I want to use Kafka connect in order to read events from Kafka topic and write them into RabbitMQ.
In order to do so, I need to use the RabbitMQ sink.
Each of the events coming from Kafka should be sent to a different queue (based on some field in the event structure), which means a different routing key should be used. As far as I know, there's an option to configure a static routing key in the sink configuration. Is there any option to configure it dynamically based on the events to achieve the required behavior?
Related
I'd like to use Kafka Connect to detect changes on a Postgres DB via CDC for a group of tables, and push them as messages in one single topic, with the key as the logic key of the main table.
This will give the consumer to consume the data changes in the right order to apply them to a destination DB.
Are there Source and Sink connectors allowing me to achieve this goal?
I'm using Debezium CDC Source Connector for Postgres... which I can configure to route all the messages for all the tables into one single topic.
But then I'm not able to find a Sink connector capable to consume the messages, and write to the right table depending on the schema of the message.
There'd be no property for a specific sink connector, if that's what you're looking for
You can use Single Message Transforms to Extract parts of the record to set the outgoing topic name from a single sink connector topic
Example : https://docs.confluent.io/platform/current/connect/transforms/extracttopic.html
I only found references to MirrorMaker v2.
Can I reuse org.apache.kafka.connect.mirror.MirrorSourceConnector as if it were a "plain" Connector with Kafka as a source, or is there something else, hopefully simpler, available?
I'm trying to use KafkaConnect and (a combination of) its SMTs to simulate message routing behaviour found in other message brokers.
For example, I would like to consume from a topic, extract values from the message (either headers or payload), and route the message to another topic within the same cluster depending on the data found in the message.
Thank you
within the same cluster
Then that's what Kafka Streams or ksqlDB are for. You can import and use SMT methods directly via code, although you also need to use the Connect Converter classes to get Schema/Struct types that most of the SMT's require
While you could use MirrorMaker, intercluster relocation is not its purpose
Currently I have a sink connector which gets data from topic A and sends its to an external service.
Now I have a use case when based on some logic I should send it to topic B instead of the service.
And this logic based on the response of the target service,that will return response based on the data.
So because the data should be sent to the target system every time I couldnt use the stream api.
Is that feasible somehow?
Or should I add a kafka producer manually to my sink? If so is there any drawback?
The first option, is to create a custom Kafka Connect Single Message Transform that will implement the desired logic and possibly use ExtractTopic as well (depending on how your custom smt looks like).
The second option is to build your own consumer. For example:
Step 1: Create one more topic on top of topic A
Create one more topic, say topic_a_to_target_system
Step 2: Implement your custom consumer
Implement a Kafka Consumer that consumes all the messages from topic topic_a.
At this point, you need to instantiate a Kafka Producer and based on the logic, decide whether the topic needs to be forwarded to topic_B or to the target system (topic_a_to_target_system).
Step 3: Start Sink connector on topic_a_to_target_system
Finally start your sink connector so that it sinks the data from topic topic_a_to_target_system to your target system.
I have tried to send the information of a Kafka Connnect instance in distributed mode with one worker to a specific topic, I have the topic name in the "archive.properties" file that use when I launch the instance.
But, when I send five or more instances, I see the messages merged in all topics.
The "solution" I thought was make a map to store the relation between ID and topic but it doesn't worked
Is there an specific Kafka connect implementation to do this?
Thanks.
First, details on how you are running connect and which connector you are using will be very helpful.
Some connectors support sending data to more than one topic. For example, confluent-jdbc-sink will send each table to a separate topic. So this could be a limitation of the connector you are using.
Also depending on the connector and your use case - whether you need to run more than one connector. With the JDBC connector, you need one connector per database and it will handle all the tables. If you run two connectors on the same database and same tables, you'll get duplicates.
In short hopefully your connector has helpful documentation.
In the next release of Apache Kafka we are adding Single Message Transformations. One of the transformations can modify the target topic based on data in the event - so you can use the transformation to perform event routing.
assume that i own a kafka cluster and i ask for some clients(web apps) to send data to the Kafka , how i can make sure that the client who will create the producer to connect to my Kafka brokers will do the partition in the right way if the client will use custom partition-er ?
AFAIK it's not possible to restrict Kafka clients/cluster to use some partitioner. But if your producer is hidden behind some facade interface, you can probably check if the Key of your message has been created the right way.
Your facade can accept ProducerRecords for example. In this case you have access to the key and value fields.
https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html