ERROR MapRZKRMFinderUtils: Unable to determine ResourceManager service address from Zookeeper - scala

Question

I am getting the error below while trying to create a DataFrame from a csv file using the below command:
val auctionDataFrame=spark.read.format("csv")
.option("inferSchema",true)
.load("/apps/auctiondata.csv")
.toDF("auctionid","bid","bidtime","bidder","bidderrate","openbid","price","item","daystolive")`
20/05/06 15:27:14 WARN ZKDataRetrieval: Can not get children of /services/resourcemanager/master with error: KeeperErrorCode = NoNode for /services/resourcemanager/master
20/05/06 15:27:14 ERROR MapRZKRMFinderUtils: Unable to determine ResourceManager service address from Zookeeper at node1:5181,node2:5181,node3:5181
java.lang.RuntimeException: Unable to determine ResourceManager service address from Zookeeper at node1:5181,node2:5181,node3:5181
at org.apache.hadoop.yarn.client.MapRZKRMFinderUtils.mapRZkBasedRMFinder(MapRZKRMFinderUtils.java:121)
at org.apache.hadoop.yarn.client.MapRZKBasedRMAddressFinder.getRMAddress(MapRZKBasedRMAddressFinder.java:43)
at org.apache.hadoop.yarn.conf.HAUtil.getCurrentRMAddress(HAUtil.java:72)
at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:60)
at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:74)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:114)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:317)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:206)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1333)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.take(RDD.scala:1327)
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1368)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.first(RDD.scala:1367)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.findFirstLine(CSVFileFormat.scala:206)
at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:60)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:184)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$getOrInferFileFormatSchema(DataSource.scala:183)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:135)
... 48 elided
I run spark-shell using : /opt/mapr/spark/spark-2.1.0/bin/spark-shell
Could you please help me how to fix this error.
Thanks
Abir

Answer 1

I have faced similar issue when my spark streaming application was compiled against older version of MapR and dependencies.
But when I re submitted the Spark app by replacing some of the dependencies by version "up to date" yarn executed it
Make sure you compile time jar's version and the runtime jars are same.
That includes Spark 2.1.0,hadoop jars