sending DataStream from VM socket to Kafka and receiving on Host OS's Flink program : Deserialization Issue - apache-kafka

Question

I am sending data stream from VM to Kafka's test topic (running on host OS at 192.168.0.12 IP ) using code below
public class WriteToKafka {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Use ingestion time => TimeCharacteristic == EventTime + IngestionTimeExtractor
env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
DataStream<JoinedStreamEvent> joinedStreamEventDataStream = env
.addSource(new JoinedStreamGenerator()).assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "192.168.0.12:9092");
properties.setProperty("zookeeper.connect", "192.168.0.12:2181");
properties.setProperty("group.id", "test");
DataStreamSource<JoinedStreamEvent> stream = env.addSource(new JoinedStreamGenerator());
stream.addSink(new FlinkKafkaProducer09<JoinedStreamEvent>("test", new TypeInformationSerializationSchema<>(stream.getType(),env.getConfig()), properties));
env.execute();
}
JoinedStreamEvent is of type DataSream<Tuple3<Integer,Integer,Integer>> it basically joins 2 streams respirationRateStream and heartRateStream
public JoinedStreamEvent(Integer patient_id, Integer heartRate, Integer respirationRate) {
Patient_id = patient_id;
HeartRate = heartRate;
RespirationRate = respirationRate;
There is another Flink program that is running on Host OS trying to read the data Stream from kafka . I am using localhost here as kafka and zookeper are running on Host OS.
public class ReadFromKafka {
public static void main(String[] args) throws Exception {
// create execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("zookeeper.connect", "localhost:2181");
properties.setProperty("group.id", "test");
DataStream<String> message = env.addSource(new FlinkKafkaConsumer09<String>("test", new SimpleStringSchema(), properties));
/* DataStream<JoinedStreamEvent> message = env.addSource(new FlinkKafkaConsumer09<JoinedStreamEvent>("test",
new , properties));*/
message.print();
env.execute();
} //main
} //ReadFromKafka
I am getting output something like this
I think I need to implement deserializer of type JoinedStreamEvent. Can someone please give me an idea how should I write, the deserializer for JoinedStreamEvent of type DataSream<Tuple3<Integer, Integer, Integer>>
Please let me know if something else needs to be done.
P.S. - I thought of writing following deserializer, but I don't think it is right
DataStream<JoinedStreamEvent> message = env.addSource(new FlinkKafkaConsumer09<JoinedStreamEvent>("test",
new TypeInformationSerializationSchema<JoinedStreamEvent>() , properties));

Answer 1

I was able to receive events VM in the same format by writing a custom serializer and deserializer for both VM and host OS program as mentioned below
public class JoinSchema implements DeserializationSchema<JoinedStreamEvent> , SerializationSchema<JoinedStreamEvent> {
#Override
public JoinedStreamEvent deserialize(byte[] bytes) throws IOException {
return JoinedStreamEvent.fromstring(new String(bytes));
}
#Override
public boolean isEndOfStream(JoinedStreamEvent joinedStreamEvent) {
return false;
}
#Override
public TypeInformation<JoinedStreamEvent> getProducedType() {
return TypeExtractor.getForClass(JoinedStreamEvent.class);
}
#Override
public byte[] serialize(JoinedStreamEvent joinedStreamEvent) {
return joinedStreamEvent.toString().getBytes();
}
} //JoinSchema
Please note that you may have to write fromstring( ) method in your event-type method, as I have added below fromString( ) JoinedStreamEvent Class
public static JoinedStreamEvent fromstring(String line){
String[] token = line.split(",");
JoinedStreamEvent joinedStreamEvent = new JoinedStreamEvent();
Integer val1 = Integer.valueOf(token[0]);
Integer val2 = Integer.valueOf(token[1]);
Integer val3 = Integer.valueOf(token[2]);
return new JoinedStreamEvent(val1,val2,val3);
} //fromstring
Events were sent from VM using below code
stream.addSink(new FlinkKafkaProducer09<JoinedStreamEvent>("test", new JoinSchema(), properties));
Events were received using following code
public static void main(String[] args) throws Exception {
// create execution environment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("zookeeper.connect", "localhost:2181");
properties.setProperty("group.id", "test");
DataStream<JoinedStreamEvent> message = env.addSource(new FlinkKafkaConsumer09<JoinedStreamEvent>("test",
new JoinSchema(), properties));
message.print();
env.execute();
} //main