Unpersist after checkpoint - scala

Question

I have written a few custom-built graph algorithms using Apache Spark Graphx. I have three queries regarding caching and checkpoint methods. As I am new to spark and graphx, I will highly appreciate a detailed explanation.
Query 1:
I am working on a large graph with 200m vertices and 350m edges. As the graph size is large, I call the checkpoint method at regular intervals on the interim graph created in the long-running graphx algorithms. The majority of the time, the checkpoint works fine. However, I have observed the following warnings in the driver logs a few times.
22/08/04 03:25:48 [WARN] o.a.s.r.ReliableCheckpointRDD - Error writing partitioner org.apache.spark.HashPartitioner#e10 to file:/mnts/nfsvol/AML/tmp/checkpoint/a153fc7b-1be7-460d-a41b-f17c820c3bf1/rdd-496
22/08/04 03:29:35 [WARN] o.a.s.r.ReliableCheckpointRDD - Error writing partitioner org.apache.spark.HashPartitioner#e10 to file:/mnts/nfsvol/AML/tmp/checkpoint/a153fc7b-1be7-460d-a41b-f17c820c3bf1/rdd-482
How to fix such warnings?
Query 2:
It is recommended to cache/persist the object (in my case, the graph object) before checkpointing so that the DAG is not recomputed during the checkpoint. However, is it safe to unpersist the object immediately after the checkpoint is over and release the memory?
FYI, I have compared both the approaches of calling and not calling the unpersist and have not found any difference in the performance. Also, in the Spark UI, I have observed that in both the approaches, the stage starts with the ReliableCheckpointRDD.
Query 3:
If the answer to query # 2 is yes, then is it risky to get the warning mentioned in Query # 1? Would it fail the job altogether?
During one of the executions of the algorithm, I received the following error that failed the job. I wonder what is the cause of this error if the answer to query # 2 is no.
22/08/04 03:55:46 [INFO] i.c.s.a.a.AMLGraphMain$ - Execution of AML Graph application failed or stopped with exception: Error while executing AML Graph Application: Error while executing AML Graph Application: Error occurred during execution of graph algorithms: Job aborted due to stage failure: Task creation failed: java.io.FileNotFoundException: File file:/mnts/nfsvol/AML/tmp/checkpoint/a153fc7b-1be7-460d-a41b-f17c820c3bf1/rdd-482/part-00000 does not exist
java.io.FileNotFoundException: File file:/mnts/nfsvol/AML/tmp/checkpoint/a153fc7b-1be7-460d-a41b-f17c820c3bf1/rdd-482/part-00000 does not exist
      at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
      at org.apache.spark.rdd.ReliableCheckpointRDD.org$apache$spark$rdd$ReliableCheckpointRDD$$getPartitionBlockLocations(ReliableCheckpointRDD.scala:102)
      at org.apache.spark.rdd.ReliableCheckpointRDD.getPreferredLocations(ReliableCheckpointRDD.scala:117)
      at org.apache.spark.rdd.RDD.$anonfun$preferredLocations$1(RDD.scala:299)
      at org.apache.spark.rdd.RDD$$Lambda$1728/00000000B4035EF0.apply(Unknown Source)
      at scala.Option.map(Option.scala:230)
      at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:299)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2134)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$3(DAGScheduler.scala:2145)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1734/00000000B4037B50.apply$mcVI$sp(Unknown Source)
      at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2(DAGScheduler.scala:2144)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$getPreferredLocsInternal$2$adapted(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$1733/00000000B4036AD0.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocsInternal(DAGScheduler.scala:2142)
      at org.apache.spark.scheduler.DAGScheduler.getPreferredLocs(DAGScheduler.scala:2108)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitMissingTasks$2(DAGScheduler.scala:1166)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitMissingTasks$2$adapted(DAGScheduler.scala:1166)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$2427/00000000B43AEE30.apply(Unknown Source)
      at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
      at scala.collection.TraversableLike$$Lambda$26/00000000A52C4560.apply(Unknown Source)
      at scala.collection.Iterator.foreach(Iterator.scala:941)
      at scala.collection.Iterator.foreach$(Iterator.scala:941)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
      at scala.collection.IterableLike.foreach(IterableLike.scala:74)
      at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
      at scala.collection.TraversableLike.map(TraversableLike.scala:238)
      at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
      at scala.collection.AbstractTraversable.map(Traversable.scala:108)
      at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1166)
      at org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1118)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5(DAGScheduler.scala:1121)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5$adapted(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$2420/00000000B43ABF50.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5(DAGScheduler.scala:1121)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5$adapted(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$2420/00000000B43ABF50.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5(DAGScheduler.scala:1121)
      at org.apache.spark.scheduler.DAGScheduler.$anonfun$submitStage$5$adapted(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler$$Lambda$2420/00000000B43ABF50.apply(Unknown Source)
      at scala.collection.immutable.List.foreach(List.scala:392)
      at org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1120)
      at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1061)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2196)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
: Array(in.co.sbi.analytics.aml.AMLGraphMain$.executeGraphApp$1(aml_graphx_main.scala:622), in.co.sbi.analytics.aml.AMLGraphMain$.main(aml_graphx_main.scala:642), in.co.sbi.analytics.aml.AMLGraphMain.main(aml_graphx_main.scala), sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method), sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62), sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.lang.reflect.Method.invoke(Method.java:498), org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52), org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928), org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180), org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203), org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90), org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007), org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016), org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala))
Here are the cluster details: 32 executors with 4 CPU and 24GB memory each.
Kindly let me know if any more details are required. Thanks!