Encoding issue when streaming logs from AWS Kinesis to ElasticSearch via Logstash - encoding

Question

I've got an AWS Kinesis data stream called "otelpoc".
In Logstash, I'm using the Kinesis input plugin - see here.
My Logstash config is as follows:
input {
kinesis {
kinesis_stream_name => "otelpoc"
region => "ap-southeast-2"
codec => json { }
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "otelpoc-logstash-%{+YYYY.MM.dd}"
}
}
I can put events to Kinesis using the AWS CLI as follows:
aws kinesis put-record --stream-name otelpoc --data file://payload.json --partition-key 1
payload.json looks like this:
{
"message": "this is a test",
"level": "error"
}
... but when I do this I see an error in Logstash as follows:
Received an event that has a different character encoding than you configured. {:text=>"\\x99\\xEB,j\\a\\xAD\\x86+\\\"\\xB1\\xAB^\\xB2\\xD9^\\xBD\\xE9^\\xAE\\xBA+", :expected_charset=>"UTF-8"}
Interestingly the message still get's outputted to Elastic and I can view it in Kibana as shown below:
Not sure what I should be doing with the character encoding... I've tried several things in Logstash, but no success e.g. changing the codec in the kinesis input to something like the following
codec => plain {
charset => "UTF-8"
}
... but no luck... I tried to decode the encoded text in a few online decoders, but not really sure what I'm trying to decode from... anyone able to help?
EDIT: using v6.7.1 of ELK stack, which is quite old, but I don't think this is the issue...

Answer 1

I never resolved this when publishing messages to Kinesis using the AWS CLI, but for my specific use case I was trying to send logs to Kinesis using the awskinesis exporter for the Open Telemetry (OTEL) collector agent - see here.
If I use otlp_json encoding, it worked e.g.
awskinesis:
aws:
stream_name: otelpoc
region: ap-southeast-2
encoding:
name: otlp_json