Create an scala sbt project and using spark functionality - scala

Question

I have a spark application that i want to run using sbt. If a run just an application using only scala code, it works. But when a try to import spark functionalities and perform spark code, it wont work. This is my spark script:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark._
object hi {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("hi").setMaster("local[2]");
// Create a Scala Spark Context.
val sc = new SparkContext(conf)
// Load our input data.
val file1 = sc.textFile("geotweets.tsv")
val a2 = file1.map(_.split("\t")).map(rec => rec(1)).take(10).foreach(println)
}
}
And my build.sbt is like this
name := "Spark-test"
version := "1.0"
scalaVersion := "2.10.2"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.0.2"
)
But when i run this application in sbt i get this error-message:
[info] Compiling 1 Scala source to C:\Users\kolbj\OneDrive - NTNU\Emner\BigData\SBT-Phase2\target\scala-2.10\classes ...
[info] Done compiling.
[info] Packaging C:\Users\kolbj\OneDrive - NTNU\Emner\BigData\SBT-Phase2\target\scala-2.10\faen_2.10-1.0.jar ...
[info] Done packaging.
[info] Running hi
18/04/21 15:20:37 INFO spark.SecurityManager: Changing view acls to: kolbj
18/04/21 15:20:37 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kolbj)
18/04/21 15:20:38 INFO slf4j.Slf4jLogger: Slf4jLogger started
18/04/21 15:20:38 INFO Remoting: Starting remoting
18/04/21 15:20:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark#LAPTOP-9N8CNCEL:51096]
18/04/21 15:20:38 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark#LAPTOP-9N8CNCEL:51096]
18/04/21 15:20:38 INFO spark.SparkEnv: Registering MapOutputTracker
18/04/21 15:20:38 INFO spark.SparkEnv: Registering BlockManagerMaster
18/04/21 15:20:38 INFO storage.DiskBlockManager: Created local directory at C:\Users\kolbj\AppData\Local\Temp\spark-local-20180421152038-b562
18/04/21 15:20:38 INFO storage.MemoryStore: MemoryStore started with capacity 273.3 MB.
18/04/21 15:20:38 INFO network.ConnectionManager: Bound socket to port 51099 with id = ConnectionManagerId(LAPTOP-9N8CNCEL,51099)
18/04/21 15:20:38 INFO storage.BlockManagerMaster: Trying to register BlockManager
18/04/21 15:20:38 INFO storage.BlockManagerInfo: Registering block manager LAPTOP-9N8CNCEL:51099 with 273.3 MB RAM
18/04/21 15:20:38 INFO storage.BlockManagerMaster: Registered BlockManager
18/04/21 15:20:38 INFO spark.HttpServer: Starting HTTP Server
18/04/21 15:20:38 INFO server.Server: jetty-8.1.14.v20131031
18/04/21 15:20:38 INFO server.AbstractConnector: Started SocketConnector#0.0.0.0:51100
18/04/21 15:20:38 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.56.1:51100
18/04/21 15:20:38 INFO spark.HttpFileServer: HTTP File server directory is C:\Users\kolbj\AppData\Local\Temp\spark-17906dea-b751-4fca-9c8c-bca10d06246a
18/04/21 15:20:38 INFO spark.HttpServer: Starting HTTP Server
18/04/21 15:20:38 INFO server.Server: jetty-8.1.14.v20131031
18/04/21 15:20:38 INFO server.AbstractConnector: Started SocketConnector#0.0.0.0:51101
18/04/21 15:20:38 INFO server.Server: jetty-8.1.14.v20131031
18/04/21 15:20:38 INFO server.AbstractConnector: Started SelectChannelConnector#0.0.0.0:4040
18/04/21 15:20:38 INFO ui.SparkUI: Started SparkUI at http://LAPTOP-9N8CNCEL:4040
18/04/21 15:20:39 INFO storage.MemoryStore: ensureFreeSpace(32816) called with curMem=0, maxMem=286575820
18/04/21 15:20:39 INFO storage.MemoryStore: Block broadcast_0 stored as values to memory (estimated size 32.0 KB, free 273.3 MB)
18/04/21 15:20:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/21 15:20:39 WARN snappy.LoadSnappy: Snappy native library not loaded
[error] (run-main-0) org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/C:/Users/kolbj/OneDrive - NTNU/Emner/BigData/SBT-Phase2/geotweets.tsv
[error] org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/C:/Users/kolbj/OneDrive - NTNU/Emner/BigData/SBT-Phase2/geotweets.tsv
[error] at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
[error] at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
[error] at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:175)
18/04/21 15:20:39 ERROR spark.ContextCleaner: Error in cleaning thread
[java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:117)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:115)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:115)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1160)
at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:114)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
18/04/21 15:20:39 INFO network.ConnectionManager: Selector thread was interrupted!
18/04/21 15:20:39 ERROR util.Utils: Uncaught exception in thread SparkListenerBus
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:48)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1160)
at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:46)
error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
[error] at scala.Option.getOrElse(Option.scala:120)
[error] at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
[error] at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
[error] at scala.Option.getOrElse(Option.scala:120)
[error] at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
[error] at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
[error] at scala.Option.getOrElse(Option.scala:120)
[error] at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
[error] at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
[error] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
[error] at scala.Option.getOrElse(Option.scala:120)
[error] at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
[error] at org.apache.spark.rdd.RDD.take(RDD.scala:983)
[error] at hi$.main(hw.scala:15)
[error] at hi.main(hw.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] at sbt.Run.invokeMain(Run.scala:93)
[error] at sbt.Run.run0(Run.scala:87)
[error] at sbt.Run.execute$1(Run.scala:65)
[error] at sbt.Run.$anonfun$run$4(Run.scala:77)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] at sbt.util.InterfaceUtil$$anon$1.get(InterfaceUtil.scala:10)
[error] at sbt.TrapExit$App.run(TrapExit.scala:252)
[error] at java.lang.Thread.run(Thread.java:748)
[error] java.lang.RuntimeException: Nonzero exit code: 1
[error] at sbt.Run$.executeTrapExit(Run.scala:124)
[error] at sbt.Run.run(Run.scala:77)
[error] at sbt.Defaults$.$anonfun$bgRunTask$5(Defaults.scala:1172)
[error] at sbt.Defaults$.$anonfun$bgRunTask$5$adapted(Defaults.scala:1167)
[error] at sbt.internal.BackgroundThreadPool.$anonfun$run$1(DefaultBackgroundJobService.scala:366)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] at scala.util.Try$.apply(Try.scala:209)
[error] at sbt.internal.BackgroundThreadPool$BackgroundRunnable.run(DefaultBackgroundJobService.scala:289)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
sbt:FAEN> [error] (Compile / run) Nonzero exit code: 1
[error] Total time: 16 s, completed 21.apr.2018 15:20:39
18/04/21 15:20:42 INFO storage.BlockManager: Removing broadcast 0
18/04/21 15:20:42 INFO spark.ContextCleaner: Cleaned broadcast 0
18/04/21 15:20:42 INFO storage.BlockManager: Removing block broadcast_0
18/04/21 15:20:42 INFO storage.MemoryStore: Block broadcast_0 of size 32816 dropped from memory (free 286575820)
i know that the spark code works fine when using spark REPL. Also this spark code needs to retreive a tsv file using this line
val file1 = sc.textFile("geotweets.tsv")
So my second question is where should this file be placed?
My project repository is like this:
SBT-phase2(project name)
\build.sbt
\src\main\scala\hw.scala
\src\main\scala\geotweets.tsv
Anyone who knows how to fix this? :)

Answer 1

org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/C:/Users/kolbj/OneDrive - NTNU/Emner/BigData/SBT-Phase2/geotweets.tsv
The path of file you provided is wrong. Fix this. It will be good if you provide absolute path

Answer 2

You can use java.io.File's CanonicalPath api as
val file1 = sc.textFile(new java.io.File(".").getCanonicalFile+"\src\main\scala\geotweets.tsv")