Flink handling Kafka messages with parsing error - apache-kafka

Question

I have some Kafka messages of type InputIoTMessage coming in from Kafka and consumed through FlinkKafkaConsumer as below. I want to add an error field in InputIoTMessage class if there is a NoSuchFieldException. Also, Is this the best practice to handle this types of scenario or we have something more elegant in Java 8 e.g. using Option or Future?
String inputTopic = "sensors";
String outputTopic = "sensors_out";
String consumerGroup = "baeldung";
String address = "kafka:9092";
StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
FlinkKafkaConsumer011<InputIoTMessage> flinkKafkaConsumer = createIoTConsumerForTopic(inputTopic, address, consumerGroup);
flinkKafkaConsumer.setStartFromEarliest();
DataStream<InputIoTMessage> stringInputStream = environment.addSource(flinkKafkaConsumer);
System.out.println("IoT Message received :: " );
stringInputStream
.filter((event) -> {
if(event.has("jsonParseError")) {
LOG.warn("JsonParseException was handled: " + event.get("jsonParseError").asText());
return false;
}
return true;
})
.print();
InputIoTMessage.java (has method to check if field exists)
public boolean has(String fieldName) {
boolean isExists;
try {
isExists = fieldName.equalsIgnoreCase(this.getClass().getField(fieldName).getName());
} catch (NoSuchFieldException | SecurityException e) {
Field[] fieldArr = this.getClass().getDeclaredFields();
//Question: how to add "jsonParseError" field to the object here ?
}
return true;
}

Answer 1

The filter function does not modify the input records, maybe you can implement the flatMap function, after modifying the record, output through out.collect
stringInputStream.flatMap(new FlatMapFunction<InputIoTMessage, InputIoTMessage>() {
#Override
public void flatMap(InputIoTMessage input, Collector<InputIoTMessage> out) {
if (!input.has("jsonParseError")) {
InputIoTMessage output = xxxxx;
out.collect(output);
}
}
});