MQTT converter trouble to perform tansformation on values - apache-kafka

Question

I have been struggling for some days to convert values from ByteArray coming from a Mqtt Source Connector to a String. Our standalone configuration has the following parameters:
Our standalone properties file looks like this:
# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
# it to
key.converter.schemas.enable=true
value.converter.schemas.enable=true
When we try to perform some transformation on the values, it looks like Kafka has data format as ByteArrays, therefore it is not possible to perform any operation on the value. Is there a way to convert the ByteArray to a StringConverter?
What we tried to do is to change the parameters as above:
converter.encoding=UTF-8
value.converter=org.apache.kafka.connect.storage.StringConverter
We had no luck. Our Input is a string in python similar to:
{'id': 42, 'cost': 4000}
Any suggestion how to configure the property file?
EDIT: As required, I am providing more information, we moved to distributed mode, with a cluster of 3 brokers, we start the connector as follows:
{
"name":"MqttSourceConnector",
"config":{
"connector.class":"io.confluent.connect.mqtt.MqttSourceConnector",
"tasks.max":"2",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"mqtt.server.uri":"ssl://XXXXXX.XXXX.amazonaws.com:1883",
"mqtt.topics":"/mqtt/topic",
"mqtt.ssl.trust.store.path":"XXXXXXXXXX.jks",
"mqtt.ssl.trust.store.password":"XXXXX",
"mqtt.ssl.key.store.path":"XXXXXXXXX.jks",
"mqtt.ssl.key.store.password":"XXXXXX",
"mqtt.ssl.key.password":"XXXXXXXX",
"max.retry.time.ms":86400000,
"mqtt.connect.timeout.seconds":2,
"mqtt.keepalive.interval.seconds":4
}
What I receive as kwy value is:
/mqtt/topic:"stringified json with kafka topic and key"
What I would like to see:
Topic: kafka.topic.from.the.mqtt.paylod.string
Key: key_from_mqtt_string
Ofc the mqtt payload should be a json for us, but I can't manage to convert it

Answer 1

Single Message Transform usually require a schema in the data.
Connectors like this, and others connecting from message queues, generally require the ByteArrayConverter. After that you need to apply a schema to it after which you can start to manipulate the fields.
I wrote about one way of doing this here in which you ingest the raw bytes to a Kafka topic and then apply a schema using a stream processor (ksqlDB in my example, but you can use other option if you want).