I'm having issues with getting Scala Worksheets to work in IntelliJ.
I've already reinstalled JDK 8 and installed the latest SBT version.
The setting in IntelliJ seem fine to me, but I might missed something. Anyone knows how to resolve these issues?
Internal error: Could not initialize class com.sun.jna.platform.win32.WinBase$FILETIME
org.jetbrains.jps.incremental.scala.remote.ClientEventProcessor.process(ClientEventProcessor.scala:22)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.handle(RemoteResourceOwner.scala:47)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.handle$(RemoteResourceOwner.scala:37)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner.handle(RemoteServerRunner.scala:16)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$5(RemoteResourceOwner.scala:30)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$5$adapted(RemoteResourceOwner.scala:29)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$3(RemoteResourceOwner.scala:29)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$3$adapted(RemoteResourceOwner.scala:25)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$2(RemoteResourceOwner.scala:25)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$2$adapted(RemoteResourceOwner.scala:24)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.send(RemoteResourceOwner.scala:24)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.send$(RemoteResourceOwner.scala:22)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner.send(RemoteServerRunner.scala:16)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner$$anon$1.$anonfun$run$1(RemoteServerRunner.scala:36)
scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner$$anon$1.run(RemoteServerRunner.scala:32)
org.jetbrains.plugins.scala.worksheet.server.RemoteServerConnector.compileAndRun(RemoteServerConnector.scala:111)
org.jetbrains.plugins.scala.worksheet.processor.WorksheetCompiler$$anon$3.run(WorksheetCompiler.scala:66)
com.intellij.compiler.progress.CompilerTask.run(CompilerTask.java:192)
com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:750)
com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:157)
com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:580)
com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:525)
com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:85)
com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:144)
com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:395)
com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:305)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Settings IntelliJ:
Related
While submitting scala spark project I am getting this error
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
After many repeated attempts I am not able to solve this problem. Can anybody help me to submit my project
I have tried revalidating the cache and submitted the project still not solved
I was executing a scala program with SBT and needed to stop the execution, so I hit Ctrl+c to end execution, which also ends the execution of SBT. I've done this a thousand times, but this time SBT wouldn't restart, and gives me this error:
java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Appender
at sbt.StandardMain$.initialGlobalLogging(Main.scala:114)
at sbt.StandardMain$.initialState(Main.scala:136)
at sbt.xMain.run(Main.scala:70)
at xsbt.boot.Launch$.$anonfun$run$1(Launch.scala:149)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:176)
at xsbt.boot.Launch$.run(Launch.scala:149)
at xsbt.boot.Launch$.$anonfun$apply$1(Launch.scala:44)
at xsbt.boot.Launch$.launch(Launch.scala:159)
at xsbt.boot.Launch$.apply(Launch.scala:44)
at xsbt.boot.Launch$.apply(Launch.scala:21)
at xsbt.boot.Boot$.runImpl(Boot.scala:78)
at xsbt.boot.Boot$.run(Boot.scala:73)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.Appender
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 14 more
[error] [launcher] error during sbt launcher: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Appender
I tried installing a new version of SBT but that didn't work, and I get the same error.
I'm at a complete loss as to how to fix this problem. I really don't even know what the problem is. Thanks for any help.
I was able to get things working but I won't guarantee this would be a fix for everyone. Deleting my .sbt folder and restarting sbt worked. It created a new .sbt folder and everything is working correctly now. I only use sbt for compiling scala code so there isn't much that depends on sbt. Definitely back up your old .sbt folder if you're going to try this route.
I want to connect my locally installed zeppelin 0.10.0 to an also locally installed spark 3.2.0 (I tried the same procedure with spark2.3.0 and it worked.). But it looks like zeppelin itself has an internal spark which uses the internal one every time I try. I have gone through the setting for spark interpreters with no use.
I just want to know if there is anyway I can change the default internal spark that zeppelin uses and change it to a spark 3.2.0 I want to use.
I put the parameters of SPARK_HOME what it is said to be and spark.master local[*] receiving the following error:
org.apache.zeppelin.interpreter.InterpreterException: java.lang.NoSuchMethodError: scala.tools.nsc.Settings.usejavacp()Lscala/tools/nsc/settings/AbsSettings$AbsSetting;
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:833)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:741)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: scala.tools.nsc.Settings.usejavacp()Lscala/tools/nsc/settings/AbsSettings$AbsSetting;
at org.apache.zeppelin.spark.SparkScala212Interpreter.open(SparkScala212Interpreter.scala:66)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:121)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more
org.apache.zeppelin.interpreter.InterpreterException: java.lang.NoSuchMethodError: scala.tools.nsc.Settings.usejavacp()Lscala/tools/nsc/settings/AbsSettings$AbsSetting;
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:833)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:741)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: scala.tools.nsc.Settings.usejavacp()Lscala/tools/nsc/settings/AbsSettings$AbsSetting;
at org.apache.zeppelin.spark.SparkScala212Interpreter.open(SparkScala212Interpreter.scala:66)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:121)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more
I've run into the same issue myself - you won't run Spark 3.2.0 on Zeppelin 0.10.0. Spark 3.1.2 works without any issues and Zeppelin has Spark 2.4.5 included - this is a problem with a tool itself.
According to the ticket ZEPPELIN-5565 version 0.10.0 DOES NOT support Spark 3.2.0. This should be fixed in 0.10.1 and 0.11.0 (info from mentioned ticket and I've also checked the Github repo).
Pull request that fixes this issue is much longer, but in Zeppelin 0.10.0 there is this strategic line:
public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_3_2_0;
I can't define function in scala worksheet in Intelijj IDEA. Also worksheet doesn't show anything when in repl run type - I have to switch to plain run type. I can define for example variables in worksheet. When I run scala repl from terminal I can define function as expected.
Running on Ubuntu 18.04. I'm using Intelijj IDEA 2019.1.3, I've installed Java JDK 8.0_221 (with instructions from https://www.vultr.com/docs/how-to-manually-install-java-8-on-ubuntu-16-04 page) and scala 2.13.0 from .deb package from scala page.
in a worksheet:
def max(x: Int, y: Int): Int = {
if(x > y) x
else y
}
Expected:
function max is defined
Actual result:
Internal error: java.lang.NoClassDefFoundError: scala/Serializable
org.jetbrains.jps.incremental.scala.remote.ClientEventProcessor.process(ClientEventProcessor.scala:22)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.handle(RemoteResourceOwner.scala:47)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.handle$(RemoteResourceOwner.scala:37)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner.handle(RemoteServerRunner.scala:16)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$5(RemoteResourceOwner.scala:30)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$5$adapted(RemoteResourceOwner.scala:29)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$3(RemoteResourceOwner.scala:29)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$3$adapted(RemoteResourceOwner.scala:25)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$2(RemoteResourceOwner.scala:25)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.$anonfun$send$2$adapted(RemoteResourceOwner.scala:24)
org.jetbrains.jps.incremental.scala.package$.using(package.scala:21)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.send(RemoteResourceOwner.scala:24)
org.jetbrains.jps.incremental.scala.remote.RemoteResourceOwner.send$(RemoteResourceOwner.scala:22)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner.send(RemoteServerRunner.scala:16)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner$$anon$1.$anonfun$run$1(RemoteServerRunner.scala:36)
scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:155)
org.jetbrains.plugins.scala.compiler.RemoteServerRunner$$anon$1.run(RemoteServerRunner.scala:32)
org.jetbrains.plugins.scala.worksheet.server.RemoteServerConnector.compileAndRun(RemoteServerConnector.scala:111)
org.jetbrains.plugins.scala.worksheet.processor.WorksheetCompiler$$anon$3.run(WorksheetCompiler.scala:61)
com.intellij.compiler.progress.CompilerTask.run(CompilerTask.java:185)
com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:731)
com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:164)
com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:586)
com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:532)
com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:86)
com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:151)
com.intellij.openapi.progress.impl.CoreProgressManager$4.run(CoreProgressManager.java:403)
com.intellij.openapi.application.impl.ApplicationImpl$1.run(ApplicationImpl.java:311)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
I've had same problem with scala 2.13.0 and wasted a lot of time to find correct advice. Then I changed Global Libraries to scala-sdk-2.12.7, now everything works correct in REPL.
In my case, I followed the suggestion proposed in this recent post in the JetBrains issue tracker, where the solution is to set the compile server to JDK 11 (or later).
I have a problem with execution of local .jar on Zeppelin. I'm adding dependency jar via this guide, but when I go to notebook and try to execute
println("Hi")
I'm getting the stack listed below:
java.lang.NoSuchMethodError: scala.collection.immutable.$colon$colon.hd$1()Ljava/lang/Object;
at scala.tools.nsc.settings.MutableSettings.loop$1(MutableSettings.scala:64)
at scala.tools.nsc.settings.MutableSettings.processArguments(MutableSettings.scala:91)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:706)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
When there is no dependencies in interpreter, everything works fine.
I know that might be scala dependencies issues, but I've tried with different versions of scala and that won't help.
Also util.Properties.versionString on Zeppelin notebook returns res1: String = version 2.11.8 - it's the same version as in my test .jar file.