spark yarn cluster Stack trace: ExitCodeException exitCode=13 - scala

Question

I try to run my program in yarn cluster mode.I'm 100% sure the class in fat jar by sbt exists.
I have no clue why the spark always throw Stack trace: ExitCodeException exitCode=13 error.
And I follow the trackping page and see java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test.
Then I run the spark PI example in yarn cluster and make it.
In yarn client/local mode,it still failed with the same error：java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test
P.S. The spark conf in program looks like this:
var sparkConf = new SparkConf()
.setAppName("TiffDN2TOA")
// .setIfMissing("spark.master", masterUrl)
.set("spark.executor.memory", "10g")
.set("spark.kryoserializer.buffer.max", "1024")
implicit val sc = new SparkContext(sparkConf)
object test {
val masterUrl = "local[*]"
var sparkConf = new SparkConf()
.setAppName("TiffDN2TOA")
// .setIfMissing("spark.master", masterUrl)
.set("spark.executor.memory", "10g")
.set("spark.kryoserializer.buffer.max", "1024")
implicit val sc = new SparkContext(sparkConf)
def main(args: Array[String]): Unit = {
HadoopLandsatDN2ToaMethods.scenesDn2Toa(args(0),args(1))
}
}
spark2-submit \
> --master yarn \
> --deploy-mode cluster \
> --class org.air.ebds.organize.geotrellisETLtoa.LandsatDN2Toa \
> --num-executors 4 \
> --executor-cores 4 \
> --executor-memory 10G \
> --driver-memory 12g \
> --conf "spark.kryoserializer.buffer.max=1024m spark.kryoserializer.buffer=1024m" \
> /root/Desktop/toa.jar \
> /root/Desktop/ebds_landsat8/LC08/122/031/LC08_L1TP_122031_20140727,/root/Desktop/ebds_landsat8/LC08/122/031/LC08_L1TP_122031_20140913,LC08_L1TP_122031_20141116 \
> file:///
19/07/10 16:40:21 INFO client.RMProxy: Connecting to ResourceManager at bigdataone/192.168.1.151:8032
19/07/10 16:40:21 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/07/10 16:40:21 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (65536 MB per container)
19/07/10 16:40:21 INFO yarn.Client: Will allocate AM container, with 13516 MB memory including 1228 MB overhead
19/07/10 16:40:21 INFO yarn.Client: Setting up container launch context for our AM
19/07/10 16:40:21 INFO yarn.Client: Setting up the launch environment for our AM container
19/07/10 16:40:21 INFO yarn.Client: Preparing resources for our AM container
19/07/10 16:40:21 INFO yarn.Client: Uploading resource file:/root/Desktop/toa.jar -> hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0061/toa.jar
19/07/10 16:40:33 INFO yarn.Client: Uploading resource file:/tmp/spark-0ccc5b92-4ef5-4f5e-944b-386abcbb5938/__spark_conf__3393474382108225503.zip -> hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0061/__spark_conf__.zip
19/07/10 16:40:33 INFO spark.SecurityManager: Changing view acls to: root
19/07/10 16:40:33 INFO spark.SecurityManager: Changing modify acls to: root
19/07/10 16:40:33 INFO spark.SecurityManager: Changing view acls groups to:
19/07/10 16:40:33 INFO spark.SecurityManager: Changing modify acls groups to:
19/07/10 16:40:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
19/07/10 16:40:34 INFO yarn.Client: Submitting application application_1561542066113_0061 to ResourceManager
19/07/10 16:40:34 INFO impl.YarnClientImpl: Submitted application application_1561542066113_0061
19/07/10 16:40:35 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:35 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.root
start time: 1562748034023
final status: UNDEFINED
tracking URL: http://bigdataone:8088/proxy/application_1561542066113_0061/
user: root
19/07/10 16:40:36 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:37 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:38 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:39 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:40 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:41 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:42 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:43 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:44 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:45 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:46 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:47 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:48 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:49 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:50 INFO yarn.Client: Application report for application_1561542066113_0061 (state: ACCEPTED)
19/07/10 16:40:51 INFO yarn.Client: Application report for application_1561542066113_0061 (state: FAILED)
19/07/10 16:40:51 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1561542066113_0061 failed 2 times due to AM Container for appattempt_1561542066113_0061_000002 exited with exitCode: 13
For more detailed output, check application tracking page:http://bigdataone:8088/proxy/application_1561542066113_0061/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1561542066113_0061_02_000001
Exit code: 13
Stack trace: ExitCodeException exitCode=13:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 13
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.root
start time: 1562748034023
final status: FAILED
tracking URL: http://bigdataone:8088/cluster/app/application_1561542066113_0061
user: root
Exception in thread "main" org.apache.spark.SparkException: Application application_1561542066113_0061 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1153)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1568)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:892)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/07/10 16:40:51 INFO util.ShutdownHookManager: Shutdown hook called
19/07/10 16:40:51 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-5e6eb641-9f2a-4351-947d-a3b4cf578f6d
19/07/10 16:40:51 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0ccc5b92-4ef5-4f5e-944b-386abcbb5938
The application tracking page:http://bigdataone:8088/proxy/application_1561542066113_0061 shows:
19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for TERM
19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for HUP
19/07/10 15:14:10 INFO util.SignalUtils: Registered signal handler for INT
19/07/10 15:14:10 INFO spark.SecurityManager: Changing view acls to: yarn,root
19/07/10 15:14:10 INFO spark.SecurityManager: Changing modify acls to: yarn,root
19/07/10 15:14:10 INFO spark.SecurityManager: Changing view acls groups to:
19/07/10 15:14:10 INFO spark.SecurityManager: Changing modify acls groups to:
19/07/10 15:14:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root); groups with view permissions: Set(); users with modify permissions: Set(yarn, root); groups with modify permissions: Set()
19/07/10 15:14:10 INFO yarn.ApplicationMaster: Preparing Local resources
19/07/10 15:14:11 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1561542066113_0055_000002
19/07/10 15:14:11 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
19/07/10 15:14:11 ERROR yarn.ApplicationMaster: Uncaught exception:
java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:682)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:448)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:301)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:241)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:241)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:241)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:782)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:781)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:240)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:806)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
19/07/10 15:14:11 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.lang.ClassNotFoundException: org.air.ebds.organize.geotrellisETLtoa.test)
19/07/10 15:14:11 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://bigdataone:8020/user/root/.sparkStaging/application_1561542066113_0055
19/07/10 15:14:11 INFO util.ShutdownHookManager: Shutdown hook called