I am trying to submit a job to spark on my machine as so:
$ spark-submit --master local --class ai.affable.flint.Foo target/scala-2.11/flint.jar
However, this fails with the following error:
java.lang.ClassNotFoundException: ai.affable.flint.Foo
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
I have verfied that the JAR file exists and has a class called Foo:
$ jar tvf ./target/scala-2.11/flint.jar | grep Foo
2003 Fri Dec 14 20:53:40 MYT 2018 ai/affable/flint/Foo.class
...
This baffles me because:
a) the JAR exists b) the class exists in the jar 3) I have specified the fully qualified path and double checked for any path errors or mispellings.
Does anyone know what I am missing?
EDIT:
I got it to work by recreating the project in a fresh directory.I literally copy pasted the code and repeated the steps.
I will still like to know what I can do in situations like this short of recreating the project.
Related
I am just starting out with ScalaTest and following http://www.scalatest.org/quick_start page
I followed the steps as outlined (by downloading the jar file linked above the code)
and the code fails
unittest vi ExampleSpec.scala
➜ unittest scalac -cp scalatest-app_2.13-3.0.8.jar ExampleSpec.scala
➜ unittest scala -cp scalatest-app_2.13-3.0.8.jar org.scalatest.run ExampleSpec
An exception or error caused a run to abort. This may have been caused by a problematic custom reporter.
java.lang.NoClassDefFoundError: scala/xml/NamespaceBinding
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1368)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1033)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1011)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1509)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1011)
at org.scalatest.tools.Runner$.main(Runner.scala:827)
at org.scalatest.run$.main(run.scala:120)
at org.scalatest.run.main(run.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:105)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:40)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:130)
at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:105)
at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:97)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:130)
at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:29)
at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:28)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:43)
at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:35)
at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:34)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:70)
at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:91)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:108)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.ClassNotFoundException: scala.xml.NamespaceBinding
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 29 more
My Scala Version is
➜ unittest scala -version
Scala code runner version 2.13.1 -- Copyright 2002-2019, LAMP/EPFL and Lightbend, Inc.
I thought the error is because scala-xml is a separate project than Scala, so I downloaded the jar file from https://mvnrepository.com/artifact/org.scala-lang/scala-xml/2.11.0-M4
Next, I tried the entire process again and it fails with different error
unittest scala -classpath "scalatest-app_2.13-3.0.8.jar:scala-xml-2.11.0-M4.jar" org.scalatest.run ExampleSpec
An exception or error caused a run to abort. This may have been caused by a problematic custom reporter.
java.lang.NoClassDefFoundError: scala/Serializable
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:515)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:423)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:417)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:416)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1368)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1033)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1011)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1509)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1011)
at org.scalatest.tools.Runner$.main(Runner.scala:827)
at org.scalatest.run$.main(run.scala:120)
at org.scalatest.run.main(run.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:105)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:40)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:130)
at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:105)
at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:97)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:130)
at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:29)
at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:28)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:43)
at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:35)
at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:34)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:70)
at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:91)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:108)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.ClassNotFoundException: scala.Serializable
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 39 more
Could someone please help me understand what the issue is?
Thank you
UPDATE (SOLVED)
As per the comment from #J0HN and answer from #Mario, I was able to solve the issue
➜ unittest ll
total 17608
-rw-r--r-- 1 harit staff 504B Nov 21 17:15 ExampleSpec.scala
-rw-r--r--# 1 harit staff 555K Nov 22 10:35 scala-xml_2.13-1.2.0.jar
-rw-r--r--# 1 harit staff 7.9M Nov 21 16:59 scalatest-app_2.13-3.0.8.jar
➜ unittest scalac -classpath "scalatest-app_2.13-3.0.8.jar:scala-xml_2.13-1.2.0.jar" ExampleSpec.scala
➜ unittest scala -classpath "scalatest-app_2.13-3.0.8.jar:scala-xml_2.13-1.2.0.jar" org.scalatest.run ExampleSpec
Run starting. Expected test count is: 2
ExampleSpec:
A Stack
- should pop values in last-in-first-out order
- should throw NoSuchElementException if an empty stack is popped
Run completed in 506 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
➜ unittest
I have downloaded scala-xml_2.13-1.2.0.jar from this page
Scala 2.13 decoupled scala-xml from the standard library:
The following modules are no longer included in the distribution:
scala-xml, scala-parser-combinators, scala-swing. They are
community-maintained and published to Maven Central.
As suggested by J0HN, download scala-xml_2.13 and try the following command
scala -cp scalatest-app_2.13-3.0.8.jar:scala-xml_2.13-1.2.0.jar org.scalatest.run ExampleSpec
Here's a brilliant web talk by J. A. De Goes: https://www.youtube.com/watch?v=sxudIMiOo68 - highly recommended for everyone interested in functional programming
And here's accompanying code gist:
https://gist.github.com/jdegoes/1b43f43e2d1e845201de853815ab3cb9
When I run $ scalac fpmax.scala, it compiles everything to a new directory fpmax
But then when I run scala App0, it gives me error:
Exception in thread "main" java.lang.NoClassDefFoundError: App0 (wrong name: fpmax/App0)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at scala.reflect.internal.util.ScalaClassLoader.$anonfun$tryClass$1(ScalaClassLoader.scala:45)
at scala.util.control.Exception$Catch.$anonfun$opt$1(Exception.scala:242)
at scala.util.control.Exception$Catch.apply(Exception.scala:224)
at scala.util.control.Exception$Catch.opt(Exception.scala:242)
at scala.reflect.internal.util.ScalaClassLoader.tryClass(ScalaClassLoader.scala:45)
at scala.reflect.internal.util.ScalaClassLoader.tryToLoadClass(ScalaClassLoader.scala:39)
at scala.reflect.internal.util.ScalaClassLoader.tryToLoadClass$(ScalaClassLoader.scala:39)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.tryToLoadClass(ScalaClassLoader.scala:125)
at scala.reflect.internal.util.ScalaClassLoader$.classExists(ScalaClassLoader.scala:150)
at scala.tools.nsc.GenericRunnerCommand.guessHowToRun(GenericRunnerCommand.scala:36)
at scala.tools.nsc.GenericRunnerCommand.<init>(GenericRunnerCommand.scala:55)
at scala.tools.nsc.GenericRunnerCommand.<init>(GenericRunnerCommand.scala:18)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:42)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:101)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
I run scala 2.12.6.
What would be correct way to run this code snippet?
I had to edit it a bit to run App0.
Change def main: Unit to def main(args: Array[String]): Unit, so that it has main method with args as per Scala spec.
Let's compile it:
scala fpmax.scala. Indeed, this will create fpmax directory with all classes in it
Then run the App0:
scala fpmax.App0 5.50 ✔ 12:21 19.03.19
What is your name?
Alex
Hello, Alex, welcome to the game!
Dear Alex, please guess a number from 1 to 5:
4
You guessed wrong, Alex! The number was: 3
Do you want to continue, Alex?
n
Note, I am NOT going inside the fpmax directory with classes
I am trying to run all the tests (all of them extend FlatSpec) in a jar using the ScalaTest Runner, but get a message for VerifyError. However, I am able to individually run the tests.
The jar was compiled using sbt test:assembly. I am trying to run the tests in another environment where sbt is not available. The tests are available in the jar in com/tfs/test path, as shown in the output below:
samik#samik-lap:~/git/proj$ jar tf test-2018.2.jar | grep MyTest
com/tfs/test/MyTest$$anonfun$1$$anonfun$apply$mcV$sp$1.class
com/tfs/test/MyTest$$anonfun$4.class
com/tfs/test/MyTest$$anonfun$5.class
com/tfs/test/MyTest$$typecreator4$1.class
com/tfs/test/MyTest$$typecreator5$1.class
com/tfs/test/MyTest$$typecreator9$1.class
com/tfs/test/MyTest.class
The following command runs the specific test just fine:
samik#samik-lap:~/git/proj$ scala -J-Xmx2g -cp "scalatest_2.11-3.0.5.jar:scalactic_2.11-3.0.5.jar:test-2018.2.jar" org.scalatest.run com.tfs.test.MyTest
0 [ScalaTest-main] INFO com.tfs.test.MyTest - Starting MyTest test application
Start Time: 0 sec
...
However, when I use Runner to run all the tests (there are ~7 similar tests in the jar available at the same path), it doesn't work.
samik#samik-lap:~/git/proj$ scala -J-Xmx2g -cp "scalatest_2.11-3.0.5.jar:scalactic_2.11-3.0.5.jar" org.scalatest.tools.Runner -o -R test-2018.2.jar
Discovery starting.
*** RUN ABORTED ***
java.lang.VerifyError: Cannot inherit from final class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
How do I get all the tests to run through Runner? How do I get to know what is happening inside and where the issue is? Thanks for any pointer.
With Hadoop 2.2 installed on single node I try to run Scalding tutorial, part 1, with command:
$ yarn jar target/scalding-tutorial-0.8.11.jar Tutorial0 --hdfs
https://github.com/Cascading/scalding-tutorial/
Before running tutorial I Have copied required file hello.txt to HDFS:
$ hdfs dfs -ls /data
Found 2 items
drwxr-xr-x - hdfs hdfs 0 2014-02-04 16:35 /data/10gsort
-rw-r--r-- 3 hdfs hdfs 26 2014-07-03 15:07 /data/hello.txt
It looks like tutorial can not find input file:
Exception in thread "main" com.twitter.scalding.InvalidSourceException:[TextLine(data/hello.txt)] Data is missing from one or more paths in: List(data/hello.txt)
at com.twitter.scalding.FileSource.validateTaps(FileSource.scala:102)
at com.twitter.scalding.Job$$anonfun$validateSources$1.apply(Job.scala:158)
at com.twitter.scalding.Job$$anonfun$validateSources$1.apply(Job.scala:153)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1156)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at com.twitter.scalding.Job.validateSources(Job.scala:153)
at com.twitter.scalding.Job.buildFlow(Job.scala:91)
at com.twitter.scalding.Job.run(Job.scala:126)
at com.twitter.scalding.Tool.start$1(Tool.scala:109)
at com.twitter.scalding.Tool.run(Tool.scala:125)
at com.twitter.scalding.Tool.run(Tool.scala:72)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at JobRunner$.main(JobRunner.scala:27)
at JobRunner.main(JobRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Any ideas how to make it work?
TextLine turns out to build a Hadoop Path according to the given path and configuration.
Hadoop Path API shows "A path string is absolute if it begins with a slash."
Tutorial I fixes the input to be "data/hello.txt", which actually ends up with a relative path. Current working directory will be prepended to form an absolute and solid path.
This perhaps more a Server Fault question...
I installed schema2src via sbaz and now I would like to convert a DTD (Apple's plist schema) to Scala source.
$ schema2src
usage:
schema2src [flags] --module mname arg*
or
schema2src dtd arg*
or (experimental)
schema2src xsd arg* (this doesn't work at all yet)
where supported [flags] may be:
--verbose prints some debugging information
However, if I try give any argument, it appears it can't find Scala:
$ schema2src --verbose
Exception in thread "main" java.lang.NoClassDefFoundError: scala/runtime/BoxesUtility
at schema2src.Main$.processArgs(Main.scala:56)
at schema2src.Main$.main(Main.scala:25)
at schema2src.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: scala.runtime.BoxesUtility
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
... 3 more
I have scala in my PATH...
Any suggestions? Assuming that problem can be fixed, is this the correct syntax?
$ schema2src PropertyList-1.0.dtd
I think it is now http://scalaxb.org/