Flink Kafka Connector runtime error - apache-kafka

Question

I am using:
flink 1.1.2
Kafka 2.10-0.10.0.1
flink-connector-kafka-0.9.2.10-1.0.0
I am using the following very simple/basic app
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:33334");
properties.setProperty("partition.assignment.strategy", "org.apache.kafka.clients.consumer.RangeAssignor");
properties.setProperty("group.id", "test");
String topic = "mytopic";
FlinkKafkaConsumer09<String> fkc =
new FlinkKafkaConsumer09<String>(topic, new SimpleStringSchema(), properties);
DataStream<String> stream = env.addSource(fkc);
env.execute()
After compiling it using maven and when I try to run using the following command:
bin/flink run -c com.mycompany.app.App fkaf/target/fkaf-1.0-SNAPSHOT.jar
I see the following runtime error:
Submitting job with JobID: f6e290ec7c28f66d527eaa5286c00f4d. Waiting for job completion.
Connected to JobManager at Actor[akka.tcp://flink#127.0.0.1:6123/user/jobmanager#-1679485245]
10/12/2016 15:10:06 Job execution switched to status RUNNING.
10/12/2016 15:10:06 Source: Custom Source(1/1) switched to SCHEDULED
10/12/2016 15:10:06 Source: Custom Source(1/1) switched to DEPLOYING
10/12/2016 15:10:06 Map -> Sink: Unnamed(1/1) switched to SCHEDULED
10/12/2016 15:10:06 Map -> Sink: Unnamed(1/1) switched to DEPLOYING
10/12/2016 15:10:06 Source: Custom Source(1/1) switched to RUNNING
10/12/2016 15:10:06 Map -> Sink: Unnamed(1/1) switched to RUNNING
10/12/2016 15:10:06 Map -> Sink: Unnamed(1/1) switched to CANCELED
10/12/2016 15:10:06 Source: Custom Source(1/1) switched to FAILED
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/List;)V
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.open(FlinkKafkaConsumer09.java:282)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:256)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:722)
Any idea on why the method assign() is not being found? The method is there in the
lib/kafka-clients-0.10.0.1.jar.
ParameterTool parameterTool = ParameterTool.fromArgs(args);
DataStream<String> messageStream = env.addSource(new FlinkKafkaConsumer09<String>(parameterTool.getRequired("topic"), new SimpleStringSchema(), parameterTool.getProperties()));
// print() will write the contents of the stream to the TaskManager's standard out stream
// the rebelance call is causing a repartitioning of the data so that all machines
// see the messages (for example in cases when "num kafka partitions" < "num flink operators"
messageStream.rebalance().map(new MapFunction<String, String>() {
private static final long serialVersionUID = -6867736771747690202L;
#Override
public String map(String value) throws Exception {
return "Kafka and Flink says: " + value;
}
}).print();
env.execute();

Answer 1

A NoSuchMethodError indicates a version mismatch.
I would guess the issue is that you try to connect a Kafka 0.9 consumer to a Kafka 0.10 instance. Flink 1.1.x does not provide a Kafka 0.10 consumer. However, a 0.10 consumer will be included in the upcoming 1.2.0 release.
You could try to build the Kafka 0.10 consumer yourself from the current master branch (1.2-SNAPSHOT) and use that one with Flink 1.1.2. The corresponding Flink APIs should be stable and backwards compatible from 1.2 to 1.1.