Kafka connection with Flink is not working only with YARN deployment - apache-kafka

Question

I developped a simple Flink job which consumes a Kafka stream. All is working great on local environment. But when I try to run my Flink job on my cluster with YARN, nothing is coming. (And, I don't have any error message).
./bin/flink run -m yarn-cluster -yn 4 /Flink/Flink_Stream_Kafka/jar-with-dependencies.jar --server X.X.X.X:9092 --topic logs
I have of course checked if my cluster had access to the stream and it has. I can even run my job as a simple java program on each machine of my cluster and it works.
Any insights to explain this ?
Thank you
EDIT :
object KafkaConsuming {
def main(args: Array[String]) {
// Flink parameter tool
// Allow to pass arguments to this script
val params: ParameterTool = ParameterTool.fromArgs(args)
// set up streaming execution environment
val env = StreamExecutionEnvironment.getExecutionEnvironment
// make parameters available globally
env.getConfig.setGlobalJobParameters(params)
val properties = new Properties();
properties.setProperty("bootstrap.servers", params.get("server"));
// only requied for Kafka 0.8
//properties.rsetProperty("zookeeper.connect", "X.X.X.X:2181");
val stream: DataStream[String] = env.addSource(new FlinkKafkaConsumer010[String](params.get("topic"), new SimpleStringSchema(), properties))
//stream.print()
val logs: DataStream[MinifiedLog] = stream.map(x => LogParser2.parse(x))
val sessions = logs.map { x => (x.timestamp, x.sent, 1l)}
val sessionCnt: DataStream[(Long, Long, Long)] = sessions
// key stream by sensorId
.keyBy(2)
// tumbling time window of 1 minute length
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.reduce( (x: (Long, Long, Long), y: (Long, Long, Long)) => (x._1, x._2 + y._2, x._3 + y._3))
.map { z => (z._1, z._2 / 10, z._3 / 10)}
sessionCnt.print()
env.execute("Kafka consuming")
}
}

Answer 1

The output of sessionCnt.print() will be written to the standard output on the TaskManagers.
On YARN, the easiest way to access that output is to retrieve all logs using the aggregated YARN logs (yarn logs -applicationId <appid>).
I'm not sure if the access to standard out files in the Flink UI works correctly.
Another way of checking if data arrives is using the Flink Web UI.