overloaded constructor for FlinkKafkaProducer - scala

Question

I'm trying to use Scala and Flink to publish messages into a Kafka topic. However, when creating the FlinkKafkaProducer object using the code provided in the documentation, it tells me that the contructor cannot be applied. This is the code sample:
val studentProducer = new FlinkKafkaProducer[String](
"my_topic", // target topic
new SimpleStringSchema(), // serialization schema
properties, // producer config
FlinkKafkaProducer.Semantic.EXACTLY_ONCE) // fault-tolerance
With the following imports:
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.{FlinkKafkaConsumer, FlinkKafkaProducer}
import org.apache.flink.streaming.util.serialization.SimpleStringSchema
import java.util.Properties
And this is the error I'm getting:
/home/user/Flink/flinkproj/src/main/scala/org/flink/Job.scala:83:27: overloaded method constructor FlinkKafkaProducer with alternatives:
[error] (x$1: String,x$2: org.apache.flink.streaming.connectors.kafka.KafkaSerializationSchema[String],x$3: java.util.Properties,x$4: org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic)org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer[String] <and>
[error] (x$1: String,x$2: org.apache.flink.streaming.util.serialization.KeyedSerializationSchema[String],x$3: java.util.Properties,x$4: java.util.Optional[org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner[String]])org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer[String] <and>
[error] (x$1: String,x$2: org.apache.flink.streaming.util.serialization.KeyedSerializationSchema[String],x$3: java.util.Properties,x$4: org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic)org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer[String] <and>
[error] (x$1: String,x$2: org.apache.flink.api.common.serialization.SerializationSchema[String],x$3: java.util.Properties,x$4: java.util.Optional[org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner[String]])org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer[String]
[error] cannot be applied to (String, org.apache.flink.streaming.util.serialization.SimpleStringSchema, java.util.Properties, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic)
[error] val studentProducer = new FlinkKafkaProducer[String](
The variable properties is an instance of java.util.Properties. I think it has to be in the String serializer, but I don't see what is wrong.
The details on the versions are the following, from the build.sbt:
ThisBuild / scalaVersion := "2.11.8"
val flinkVersion = "1.11.3"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-clients" % flinkVersion % "provided",
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-connector-kafka" % flinkVersion % "provided")

Answer 1

I believe the documentation is out-of-date, and you need to either provide a KafkaSerializationSchema or a KeyedSerializationSchema -- or if you use a SerializationSchema, then you need to also supply a FlinkKafkaPartitioner.
I don't happen to have an example in Scala, but here's an example in Java showing how to implement a KafkaSerializationSchema that uses an ObjectMapper to write out JSON:
/**
* A Kafka {#link KafkaSerializationSchema} to serialize {#link ClickEventStatistics}s as JSON.
*
*/
public class ClickEventStatisticsSerializationSchema implements KafkaSerializationSchema<ClickEventStatistics> {
private static final ObjectMapper objectMapper = new ObjectMapper();
private String topic;
public ClickEventStatisticsSerializationSchema(){
}
public ClickEventStatisticsSerializationSchema(String topic) {
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(
final ClickEventStatistics message, #Nullable final Long timestamp) {
try {
//if topic is null, default topic will be used
return new ProducerRecord<>(topic, objectMapper.writeValueAsBytes(message));
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("Could not serialize record: " + message, e);
}
}
}