Model JSON input data for JDBC Sink Connector - apache-kafka

Question

My goal is to process an input stream that contains DB updates coming from a legacy system and replicate it into a different DB. The messages have the following format:
Insert - The after struct contains the state of the table entry after the insert operation is complete. No before struct present;
Update - The before struct contains the state of the table entry before the update operation and the after struct only lists the modified fields along with the table key;
Delete - The before struct contains the state of the table entry before the delete operation and the after struct is not present in the message;
Examples:
Insert
{"table":"SCHEMA.TEST_TABLE","op_type":"I","after":{"id_test":8934,"test_value_1":26910,"test_value_2":"XYZ","test_value_3":"2023-01-12T18:32:13.781217"}}
Update
{"table":"SCHEMA.TEST_TABLE","op_type":"U","before":{"id_test":8934,"test_value_1":26910,"test_value_2":"XYZ","test_value_3":"2023-01-12T18:32:13.781217"},"after":{"id_test":8934,"test_value_1":null,"test_value_3":"2023-01-12T18:32:18.787337"}}
Delete
{"table":"SCHEMA.TEST_TABLE","op_type":"D","before":{"id_test":8934,"test_value_1":1499,"test_value_2":"XYZ","test_value_3":"2023-01-12T18:32:18.787337"}}
After struggling a lot with trying to deal with the lack of a schema in the input messages, I turned to kSQL to overcome this and was able - not sure if by the best approach - to get ksql to generate a output topic that can be processed by a Kafka JDBC Sink Connector using a JSON_SR format.
The problem is that the output format I get in the output topic from ksql does not allow for a correct interpretation of all the scenarios - Only the Insert scenario is correctly interpreted.
The flow in ksql was:
Process input topic to map it as a JSON object by defining a schema. Created a stream where the input object is defined TEST_TABLE_INPUT):
CREATE OR REPLACE STREAM TEST_TABLE_INPUT (
"table" VARCHAR,
op_type VARCHAR,
after STRUCT<
id_test INTEGER, test_value_1 INTEGER, test_value_2 VARCHAR, test_value_3 VARCHAR
>,
before STRUCT<
id_test INTEGER
>
) WITH (kafka_topic='input_topic', value_format='JSON');
Process TEST_TABLE_INPUT to flatten the input structures (tried using the BEFORE->* and AFTER->* strategy instead but got a Caused by: line 3:11: mismatched input '*' expecting ... error):
CREATE OR REPLACE STREAM TEST_TABLE_INTERNAL
AS SELECT
OP_TYPE,
BEFORE->id_test AS B_id_test,
AFTER->id_test AS A_id_test,
AFTER
FROM TEST_TABLE_INPUT;
With this the output stream is created:
CREATE OR REPLACE STREAM TEST_TABLE_UPDATES
WITH (
VALUE_FORMAT='JSON_SR',
KEY_FORMAT='JSON_SR',
KAFKA_TOPIC='T_TEST_UPDATES'
)
AS SELECT
COALESCE(B_id_test, A_id_test) AS id_test,
AFTER->test_value_1 AS test_value_1,
AFTER->test_value_2 AS test_value_2,
AFTER->test_value_3 AS test_value_3
FROM TEST_TABLE_INTERNAL
PARTITION BY COALESCE(B_id_test, A_id_test)
;
When processing the example input topic messages below
{"table":"SCHEMA.TEST_TABLE","op_type":"I","after":{"id_test":0,"test_value_1":17315,"test_value_2":"XYZ","test_value_3":"2023-01-12T19:06:49.694383"}}
{"table":"SCHEMA.TEST_TABLE","op_type":"U","before":{"id_test":0,"test_value_1":17315,"test_value_2":"XYZ","test_value_3":"2023-01-12T19:06:49.694383"},"after":{"id_test":0,"test_value_1":null,"test_value_3":"2023-01-12T19:06:54.702107"}}
{"table":"SCHEMA.TEST_TABLE","op_type":"D","before":{"id_test":0,"test_value_1":28605,"test_value_2":"XYZ","test_value_3":"2023-01-12T19:06:54.702107"}}
{"table":"SCHEMA.TEST_TABLE","op_type":"I","after":{"id_test":1,"test_value_1":15601,"test_value_2":"KLM","test_value_3":"2023-01-12T19:07:04.716303"}}
{"table":"SCHEMA.TEST_TABLE","op_type":"U","before":{"id_test":1,"test_value_1":15601,"test_value_2":"KLM","test_value_3":"2023-01-12T19:07:04.716303"},"after":{"id_test":1,"test_value_1":null,"test_value_3":"2023-01-12T19:07:09.723473"}}
{"table":"SCHEMA.TEST_TABLE","op_type":"I","after":{"id_test":2,"test_value_1":13386,"test_value_2":"ABC","test_value_3":"2023-01-12T19:14:23.633204"}}
to simulate the I, I-U and I-U-D scenarios the output I get in KSQL is:
ksql> SELECT * FROM TEST_TABLE_UPDATES EMIT CHANGES;
+----------+-------------+--------------+----------------------------+
|ID_TEST |TEST_VALUE_1 |TEST_VALUE_2 |TEST_VALUE_3 |
+----------+-------------+--------------+----------------------------+
|0 |17315 |XYZ |2023-01-12T19:06:49.694383 |
|0 |null |null |2023-01-12T19:06:54.702107 |
|0 |null |null |null |
|1 |15601 |KLM |2023-01-12T19:07:04.716303 |
|1 |null |null |2023-01-12T19:07:09.723473 |
|2 |13386 |ABC |2023-01-12T19:14:23.633204 |
And outside of ksQL:
kafka-console-consumer.sh --property print.key=true --topic TEST_TABLE_UPDATES --from-beginning --bootstrap-server localhost:9092
0 {"TEST_VALUE_1":17315,"TEST_VALUE_2":"XYZ","TEST_VALUE_3":"2023-01-12T19:06:49.694383"}
0 {"TEST_VALUE_1":null,"TEST_VALUE_2":null,"TEST_VALUE_3":"2023-01-12T19:06:54.702107"}
0 {"TEST_VALUE_1":null,"TEST_VALUE_2":null,"TEST_VALUE_3":null}
1 {"TEST_VALUE_1":15601,"TEST_VALUE_2":"KLM","TEST_VALUE_3":"2023-01-12T19:07:04.716303"}
1 {"TEST_VALUE_1":null,"TEST_VALUE_2":null,"TEST_VALUE_3":"2023-01-12T19:07:09.723473"}
2 {"TEST_VALUE_1":13386,"TEST_VALUE_2":"ABC","TEST_VALUE_3":"2023-01-12T19:14:23.633204"}
On top of this the Connector with the configuration as mentioned below is instantiated:
{
"name": "t_test_updates-v2",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"errors.log.enable": true,
"errors.log.include.messages": true,
"topics": "TEST_TABLE_UPDATES",
"key.converter": "io.confluent.connect.json.JsonSchemaConverter",
"key.converter.schemas.enable": false,
"key.converter.schema.registry.url": "http://schema-registry:8081",
"value.converter": "io.confluent.connect.json.JsonSchemaConverter",
"value.converter.schemas.enable": false,
"value.converter.schema.registry.url": "http://schema-registry:8081",
"connection.url": "jdbc:oracle:thin:#database:1521/xe",
"connection.user": "<REDACTED>",
"connection.password": "<REDACTED>",
"dialect.name": "OracleDatabaseDialect",
"insert.mode": "upsert",
"delete.enabled": true,
"table.name.format": "TEST_TABLE_REPL",
"pk.mode": "record_key",
"pk.fields": "ID_TEST",
"auto.create": false,
"error.tolerance": "all"
}
}
Which processes the topic - apparently - without a problem but the output in the DB does not match the expected result:
SQL> select * from test_table_repl;
ID_TEST TEST_VALUE_1 TEST_VALUE_2 TEST_VALUE_3
__________ _______________ __________________ _____________________________
1 2023-01-12T19:07:09.723473
0
2 13386 ABC 2023-01-12T19:14:23.633204
I would expect the DB query to return:
ID_TEST TEST_VALUE_1 TEST_VALUE_2 TEST_VALUE_3
__________ _______________ __________________ _____________________________
1 KLM 2023-01-12T19:07:09.723473
2 13386 ABC 2023-01-12T19:14:23.633204
As it can be seen, only the insert scenario is correctly working.
I cannot understand what I am doing wrong.
Is it the data format, the connector configuration, both or another thing I am missing?