Set kafka stream header from Int to Array[Byte] conversion - scala

Question

I'm using kafka streams on a project (using Scala) in which I need to include some Headers on the records that I'm sending to a topic. Specifically, a Header requires you to convert the value that you want to put on the Header (in my case Int and Long) to a Array[Byte]. If I try to convert an Int to a Array[Byte] using toByte function, I cannot recover the header from the topic (it appears as null). The only way to do it seems to be to convert it to a String and then use getBytes to store it on the header:
val counter: Int = 1
// counter_1 doesn't appear
context.headers().add("counter_1", Array[Byte](counter.toByte))
// counter_2 doesn't appear
context.headers().add("counter_2", ByteBuffer.allocate(4).putInt(counter).array())
// the value 1 appears on the header counter_3.
context.headers().add("counter_3", String.valueOf(counter).getBytes(StandardCharsets.UTF_8))
It doesn't seem to me that converting an int to a String is the correct way to do it. What's the right way to do it?
I'm recovering the records using the kafkacat command with the %h option. So I would get something like this:
...counter_1=,counter_2=,counter_3=1,....
Any help would be greatly appreciated.

Answer 1

Notice that Kafka Streams sends a stream of bytes without telling the Kafka topic whether these bytes are an Integer, Double, String or other type. Kafka is not aware of sent data type, it just gets some bytes.
When you fetch messages using Kafkacat it assumes the messages are Strings. So if you have sent a String encoded to bytes it is able to display it. However, when you send Integers encoded to bytes - then most likely these integers are converted to bytes representing ASCII codes of unprintable characters and that's why Kafkacat won't display them