I cloned sparkling droplet project from 'https://github.com/h2oai/h2o-droplets/tree/master/sparkling-water-droplet'. And cleaned and build project using ./gradlew clean , ./gradlew build respectively. After that tried to run project using command
spark-submit --class water.droplets.SparklingWaterDroplet build/libs/sparkling-water-droplet-app.jar
Then got following error message:
Exception in thread "main" java.lang.NoClassDefFoundError: water/fvec/Frame
at water.droplets.SparklingWaterDroplet.main(SparklingWaterDroplet.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: water.fvec.Frame
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
You have to run ./gradlew shadowJar and use the jar created by it build/libs/sparkling-water-droplet-app.jar instead since it creates a fat jar which contains all classes required while submitting a job (the build task does not do that).
Related
I'm trying to use target data-validator, but when I run it in cmd, it shows me an error like:
D:\Spark\DemoTime>spark-submit --master local data-validator_2.11-0.12.1.jar --help
21/12/28 15:05:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class.
at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:972)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:492)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
But if I give the main class name, it shows like:
D:\Spark\DemoTime>spark-submit --class com.target.data_validator.Main data-validator_2.11-0.13.0.jar --help
Exception in thread "main" java.lang.NoClassDefFoundError: com/typesafe/scalalogging/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.typesafe.scalalogging.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 13 more
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I'm following the data-validator tool from target:
https://github.com/target/data-validator
But I'm unable to sort this thing out.
You can find the jar file at:
https://github.com/target/data-validator/releases
Hence I would request your help in running this tool.
Please let me know if you need anything else.
Thanks.
From the GitHub's README:
Assemble fat jar: sbt clean assembly
spark-submit --master local data-validator-assembly-0.13.0.jar --help
You need a fat jar of data-validator + its dependencies to be able to run it. You've only provided the data-validator simple jar.
When I run my code in Eclipse, it runs smoothly. When I run it as a runnable jar file I get the following error. The code reads a string and performs a certain action if a particular substring is parsed. No external libraries are used to do this. The code runs perfectly when run on Eclipse.
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.RuntimeException: Unrecognised token /
at soft.globulation.proteinParser$SubParser.parse(proteinParser.java:82)
at soft.globulation.proteinParser.getproteinFromString(proteinParser.java:61)
at asimov.players.iravan.donkey.RobustDynamicoptimization.getConstantprotein(RobustDynamicoptimization.java:242)
at asimov.players.iravan.donkey.RobustDynamicoptimization.setup(RobustDynamicoptimization.java:97)
at bio.globulator.setup(globulator.java:80)
at bio.simple.Simpleglobulator.setup(Simpleglobulator.java:65)
at bio.glabulutionState.setup(glabulutionState.java:391)
at bio.simple.SimpleglabulutionState.startFresh(SimpleglabulutionState.java:49)
at bio.glabulutionState.run(glabulutionState.java:438)
at bio.glabulve.main(glabulve.java:769)
I tried to implement a simple mapreduce job through scala. However, when I run the package using the command,
hadoop jar hadoop.jar mapreduce.MaxTemperature hdfs://sandbox/user/ajay/input hdfs://sandbox/user/ajay/output
I get the error,
16/09/06 16:06:12 INFO mapreduce.Job: Task Id : attempt_1473177830264_0002_m_000001_2, Status : FAILED Error: java.lang.ClassNotFoundException: scala.Predef$
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at mapreduce.MaxTemperatureMapper.map(MaxTemperatureMapper.scala:17)
at mapreduce.MaxTemperatureMapper.map(MaxTemperatureMapper.scala:9)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Even though I've added the scala-library to my class path, I get the above error.
hadoop version: Hadoop 2.7.1.2.3.0.0-255
scala version: 2.11.8
java version 1.7.0_85
Any suggestion is appreciated.
Apart from adding the scala library to the client's classpath, it has to be added to all nodes where the task gets executed. This can be achieved using the ToolRunner hadoop jar scala-2.11/hadoop_2.11-0.1.0.jar mapreduce.WordCount -libjars /usr/lib/scala-2.11.8/lib/scala-library.jar
I'm currently having trouble running Junit tests created in Eclipse on IntelliJ.
My IntelliJ SDK is 1.8, the SDK of the project is 1.7, however, I'm not using any new features.
I've imported the project (properly, I hope) and the JUnit test along with the JDK is being shown in the "External Libraries" tab.
When I run my test I get an "initializationError", and when right clicking the "tests" directory and selecting "Run 'All Tests'" I get the error "No tests were found".
These are the Errors I get:
Exception in thread "main" java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.intellij.junit4.JUnit46ClassesRequestBuilder.getClassesRequest(JUnit46ClassesRequestBuilder.java:50)
at com.intellij.junit4.JUnit4TestRunnerUtil.buildRequest(JUnit4TestRunnerUtil.java:91)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:95)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.ClassNotFoundException: org.hamcrest.SelfDescribing
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
Is there anything I can do?
You need the library hamcrest-core-1.3 on your class-path. See Getting Started
I just installed Maven on Eclipse. If I run the pom.xml, I get the error below.
People who use Windows don't get this error. I have Mac OS X.
When I add slf4j-simple-1.7.5.jr and slf4j-api-1.7.5.jar to the build path, it works.
Is there a way NOT to add external jars?
EDIT 2:
I used Maven clean via Terminal and that worked. I think the problem is Maven internal in Eclipse.
EDIT 3:
I deleted the projects and checked out it all from SVN and it worked somehow.
[INFO] Scanning for projects...
[ERROR] Internal error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException: org/slf4j/LoggerFactory: org.slf4j.LoggerFactory -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:168)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:534)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.eclipse.tycho.core.maven.utils.MavenCompatiblityHelper.getPluginDescriptor(MavenCompatiblityHelper.java:86)
at org.eclipse.tycho.core.maven.utils.PluginRealmHelper.execute(PluginRealmHelper.java:95)
at org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver.getDependencyMetadata(P2TargetPlatformResolver.java:144)
at org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver.setupProjects(P2TargetPlatformResolver.java:126)
at org.eclipse.tycho.core.resolver.DefaultTychoDependencyResolver.setupProject(DefaultTychoDependencyResolver.java:87)
at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.afterProjectsRead(TychoMavenLifecycleParticipant.java:77)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:273)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
... 11 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.tycho.core.maven.utils.MavenCompatiblityHelper.getPluginDescriptor(MavenCompatiblityHelper.java:68)
... 18 more
Caused by: java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at com.ning.http.client.AsyncHttpClient.<clinit>(AsyncHttpClient.java:147)
at org.sonatype.aether.connector.async.AsyncRepositoryConnector.<init>(AsyncRepositoryConnector.java:150)
at org.sonatype.aether.connector.async.AsyncRepositoryConnectorFactory.newInstance(AsyncRepositoryConnectorFactory.java:110)
at org.sonatype.aether.impl.internal.DefaultRemoteRepositoryManager.getRepositoryConnector(DefaultRemoteRepositoryManager.java:333)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:456)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:220)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:197)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:268)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:173)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:316)
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:108)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager.java:143)
... 23 more
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:244)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:230)
... 35 more
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InternalErrorException
If your code makes use of slf4j, you'll also have to add the API and one of the implementation jars to your project. So the answer is: no.