Processing data streams in kafka - apache-kafka

Question

I am trying to set up a kafka consumer to process data from Kafka streams. I was able to set up the connection to the stream and the data is visible but it's a mixture of special characters and ASCII.
I am using the inbuilt kafka console, but have also tried the python version of confluent-kafka. The only parameters that need to be followed is to use SASL_PLAINTEXT security protocol with SCRAM-SHA-256. I am open to using other methods to also parse the output (not Java if possible).
Kafka Console
bin/kafka-console-consumer.sh --bootstrap-server server:9092 \
--topic TOPIC --from-beginning --consumer.config=consumer.properties
Confluent Kafka Python
topics = "TOPIC"
conf = {
"bootstrap.servers": "server:9092",
"group.id": "group",
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanisms" : "SCRAM-SHA-256",
}
c = Consumer(conf)
c.subscribe([topics])
running = True
while running:
message = c.poll()
print(message.value())
c.close()
Output
PLE9K1PKH3S0MAY38ChangeRequest : llZYMEgVmq2CHG:Infra RequestKSUSMAINCHANGEKC-10200-FL01DATA_MISSINGCHGUSD
DATA_MISSINGDATA_MISSINGUSD
CANCEL
▒▒12SLM:Measurement"Schedule(1 = 0)USDUSD▒▒▒
l▒l▒V?▒▒▒
llZYMEgVmq
company_team team_nameTEAM###SGP000000140381PPL000002020234
Latha M▒>▒>▒ChangeRequest
hello:1234543534 cloud abcdef▒▒▒
▒Ի▒
▒▒▒
John Smithjs12345SGP000000140381▒NPPL000002020234
▒Ի▒
I am trying to parse the data on the standard output initially, but the expectation at the end is to get the parsed data in a database. Any advice would be appreciated.

Answer 1

It seems like you have your messages encoded in binary format. To print those you will need to set up a binary decoder and pass them through that. In case you have produced them using a specific schema you might also need to deserialize the objects using the Schema Registry which contains the schema for the given topic. You are looking at something in the lines of:
message_bytes = io.BytesIO(message.value())
decoder = BinaryDecoder(message_bytes)

Answer 2

As jaivalis has mentioned there appears to be a mismatch between your producers and the consumer you are using to ingest the data. Kafka Streams exposes two properties for controlling the serialization and deserialization of data that passes through the topology; default.value.serde, default.key.serde. I recommend reviewing your streams application's configuration to find a suitable deserializer for the consumer to use.
https://kafka.apache.org/documentation/#streamsconfigs
Do note however that these serdes may be overwritten by the your streams application implementation. Be sure to review your implementation as well to ensure you have found the correct serialization format.
https://kafka.apache.org/21/documentation/streams/developer-guide/datatypes.html#overriding-default-serdes