I maded a simple spark app in this repository, where sbt run always report errors as
Exception in thread "Thread-2" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /github.com/sammyne/spark-failure-with-sbt-run/target/bg-jobs/sbt_28d6c4e/target/2ab34e8d/81ecd14d/hadoop-client-api-3.3.1.jar
Environment
sbt 1.6.2
Scala 2.13.8
Spark 3.2.1
Ubuntu 20.04
Bug reproduction
Both of the following two cases trigger errors like NoSuchFileException, which are expected to run successfully.
sbt run in one shot
sbt run
Partial errors go as
Exception in thread "Thread-2" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /github.com/sammyne/spark-failure-with-sbt-run/target/bg-jobs/sbt_b7a00768/target/2ab34e8d/81ecd14d/hadoop-client-api-3.3.1.jar
The full logs can be seen here.
sbt and then run
sbt
# after entering the sbt shell
run
At this moment, it's fine with logs as here.
However, once exit the shell, I got
[info] shutting down sbt server
22/04/27 07:55:10 INFO ShutdownHookManager: Shutdown hook called
22/04/27 07:55:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-f28ecf9a-e963-459f-b54e-f1e4825ba210
22/04/27 07:55:10 ERROR Configuration: error parsing conf core-default.xml
java.nio.file.NoSuchFileException: /github.com/sammyne/spark-failure-with-sbt-run/target/bg-jobs/sbt_28d6c4e/target/2ab34e8d/81ecd14d/hadoop-client-api-3.3.1.jar
at
...
The full error can be seen as here
Related
Context:
Scala, Spark application, fat JAR, spark 3.3.0, windows 10
IDE:IntelliJ IDEA 2022.2.2
Package: it..<...>.MyPackage
MainClass : it..<...>.MyPackage.Application
Jar building: using sbt-assembly, or with Build Artifact (JAR)
Problem: in both cases, building jar with sbt or with IntelliJ
spark-submit --verbose --master local --class it.<company>.<...>.MyPackage.Application
C:\<path to jar\MyPackage.jar 10
Error: Failed to load class it..<...>.MyPackage.Application.
22/09/20 23:14:25 INFO ShutdownHookManager: Shutdown hook called
End of the story.
Notes:
Same JAR, moved to an instance of Spark running on MacOS, no problem...
Thanks for any suggestion
Lorenzo
When I hit control-C to cancel my running akka quickstart app I get the following error.
Is this normal or is there an akka config that can avoid this?
Can someone explain why this occurs?
sbt:akka-quickstart-scala> exit
[info] shutting down sbt server
[warn] Run canceled.
Exception in thread "sbt-bg-threads-1" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ForkJoinPool.externalSubmit(ForkJoinPool.java:2328)
at java.util.concurrent.ForkJoinPool.externalPush(ForkJoinPool.java:2419)
at java.util.concurrent.ForkJoinPool.execute(ForkJoinPool.java:2648)
at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:24)
at sbt.internal.BackgroundThreadPool$BackgroundRunnable.$anonfun$cleanup$1(DefaultBackgroundJobService.scala:390)
at sbt.internal.BackgroundThreadPool$BackgroundRunnable.$anonfun$cleanup$1$adapted(DefaultBackgroundJobService.scala:389)
at scala.collection.immutable.List.foreach(List.scala:392)
at sbt.internal.BackgroundThreadPool$BackgroundRunnable.cleanup(DefaultBackgroundJobService.scala:389)
at sbt.internal.BackgroundThreadPool$BackgroundRunnable.run(DefaultBackgroundJobService.scala:359)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This is a known issue that can be seen in this github issue link.
In my case, When I upgraded my sbt to 1.3.3, same issue cropped up, my akka code was not getting closed using CTRL+C when launched by sbt run. It only logs a warning message and does not stop. So, I had to downgrade the sbt version from 1.3.3 to 1.2.8 to make it work.
Please let me know if it helps!!
When I am trying to start sbt, it fails throwing an IO exception.
Version of SBT - 1.1.2
Version of Scala - 2.12.6
Error:
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading project definition from C:\Users\username\IdeaProjects\projectname\project
[info] Loading settings from build.sbt ...
[info] Set current project to example (in build file:/C:/Users/username/IdeaProjects/projectname/)
[error] server failed to start on local:sbt-server-ad238fc8ee352b69065c. java.io.IOException: Could not create lock for
\\.\pipe\sbt-server-ad238fc8ee352b69065c_lock, error 231
Analysing the source of the throw it seems error code 231 refers to Windows error:
ERROR_PIPE_BUSY
231 (0xE7)
All pipe instances are busy.
The pipe cannot be accessed because another process is using it, so try closing down IntelliJ and any other SBT server running, and try again later. Perhaps even reboot the machine, and then start a single system SBT server (not IntelliJ), and see if the issue persists.
I am getting the following exception after starting activator ui from command line.
Checking for a newer version of Activator (current version 1.2.2)...
... our current version 1.2.2 looks like the latest.
Found previous process id: 2988
FOUND REPO = activator-local # file:////C:/shakti/scala/typesafe-activator-1.2.2
/activator-1.2.2/repository
Play server process ID is 5712
[info] play - Application started (Prod)
[info] play - Listening for HTTP on /127.0.0.1:8888
[ERROR] [06/15/2014 05:46:13.851] [default-akka.actor.default-dispatcher-3] [ActorSystem(default)] Failed to download http://downloads.typesafe.com/typesafe-activator/index/v2/index-07064250cfd1cc2444f19c731541332ccf93bc13.zip: sbt.TranslatedIOException: Error opening http://downloads.typesafe.com/typesafe-activator/index/v2/index-07064250cfd1cc2444f19c731541332ccf93bc13.zip: java.net.UnknownHostException: downloads.typesafe.com
[INFO] [06/15/2014 05:46:13.860] [default-akka.actor.default-dispatcher-3] [akka://default/user/template-cache] Unable to check remote server for template updates.
[INFO] [06/15/2014 05:46:19.155] [default-akka.actor.default-dispatcher-2] [akka://default/user/home-socket-1] Firing up web socket
What could be the reason and how to fix it?
Seems it was an intermittent issue due to some network problem. It is working fine now.
I resolve this prolem using ./activator ui instead using activator ui on linux
I can't seem to find similar issues.
On OSX, play2 bin seems to run okay but using sbt causes oom errors and scala files never complete compiling
--- (Running the application from SBT, auto-reloading is enabled) ---
[info] play - Listening for HTTP on port 9000...
(Server started, use Ctrl+D to stop and go back to the console...)
[info] Compiling 15 Scala sources and 2 Java sources to
/Users/tripled153/Development/src/Foundation/playApp/target/scala-2.9.1/classes...
Java HotSpot(TM) 64-Bit Server VM warning: Exception
java.lang.OutOfMemoryError occurred dispatching signal Unknown Signal
to handler- the VM may need to be forcibly terminated
I'm running with extended memory settings but still no luck. Running in staged (prod) mode appears okay though.
rundev.sh
export SBT_OPTS="-Xms512M -Xmx3536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=512M"
sudo sbt clean run