Kafka Connect - Not working for UPDATE operation - apache-kafka

Question

I am new to Kafka-Connect source and sink. I created application to transfer Table Data from one Schema (Schema1) to another Schema (Schema2), here I used Oracle as a Database. I successfully transferred data/row for INSERT operation from Table "Schema1.Header" to Table "Schema2.Header", but not working for UPDATE operation with below mentioned config.
SOURCE Config:
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:oracle:thin:#localhost:1524:XE",
"connection.user": "USER",
"connection.password": "user1234",
"dialect.name": "OracleDatabaseDialect",
"topic.prefix": "Schema1.Header",
"incrementing.column.name": "SC_NO",
"mode": "incrementing",
"query": "SELECT * FROM (SELECT HEADER_V1.* FROM Schema1.Header HEADER_V1 INNER JOIN Schema1.LINE_V1 LINE_V1 ON HEADER_V1.SC_NO = LINE_V1.SC_NO AND LINE_V1.CLNAME_CODE ='XXXXXX' AND HEADER_V1.ITEM_TYPE = 'XXX')",
"transforms": "ReplaceField",
"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.blacklist": "col_3,col_10"
}
SINK Config:
{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:oracle:thin:#localhost:1524:XE",
"connection.user": "USER2",
"connection.password": "user21234",
"dialect.name": "OracleDatabaseDialect",
"topics": "Schema1.Header",
"table.name.format": "Schema2.Header",
"tasks.max": "1"
}
Kindly help me to fix this issue.
Note : I need to do all CRUD operations in Schema Schema1.Tables only, Using Kafka connect am transferring those data to another Schema Schema2.Tables. Newly inserted data/row got transferred but updated data/row not transferred via Kafka-Connect. What I have to do achieve this?

Answer 1

According to this blog you need to set the mode to timestamp (or better timestamp+incrementing if you want to both new and updated rows) in your source config.
In addition, you then need to specify the timestamp.column.name which should point to a timestamp column that is updated every time the row is updated.