I am getting this below error while running the test for maven. I have HADOOP_HOME, hadoop.dll file , everything is set up in my local machine and path , environmental variables for my machine. Previously it was running fine. While I had to clone a different repository I started getting this error. Can anyone help me with this.
org.apache.spark.sql.AnalysisException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:196)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
...
Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:694)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:613)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:180)
The problem is that you haven´t set the library path. Try to set the java.library.path java property with the folder where the hadoop.dll is located. If you execute your tests with maven you can set the property of the forked jvm that executes the tests with the argLine option:
mvn -DargLine="-Djava.library.path=[hadoop_dll_dir_path]"
Related
When I use maven to run tests with this command it works fine:
mvn test -f "c:\path\to\pom.xml"
But when I run the test using the "Testing Explorer" in vscode, it fails with this error:
java.lang.NoClassDefFoundError: path/to/package/ClassName
at path.to.package.ClassName.TestName(TestFileName.java:15)
Caused by: java.lang.ClassNotFoundException: path.to.package.ClassName
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 24 more
It appears that the test doesn't find the required dependencies when it runs. However, the required dependencies for the test method are available and I can see them in other folders under the project but not under the target folder.
Perhaps the question is how to specify the class path when running the tests from the vscode Testing Explorer view.
You can try to create a customized test launch configuration in the settings. There you can specify the classpaths to be appended. See: https://github.com/microsoft/vscode-java-test/wiki/Run-with-Configuration
On a remote Windows 10 system sbt failes to start with a NoClassDefFoundError:
C:\WORKBENCH\BPF\my-project>sbt -Djavax.net.ssl.trustStoreType=WINDOWS-ROOT -Dsbt.boot.credentials="C:\WORKBENCH\BPF\my-project\credentials.txt" -Dsbt.boot.directory=C:\WORKBENCH\BPF\.sbt\boot -Dsbt.coursier.home=C:\WORKBENCH\BPF\coursier -v
# Executing command line:
"C:\Program Files (x86)\Java\jre1.8.0_321\bin\java.exe"
-Djavax.net.ssl.trustStore="C:\Program Files (x86)\Java\jre1.8.0_321\lib\security\cacerts"
-Djavax.net.ssl.trustStorePassword=changeit
-Xms1024m
-Xmx1024m
-Xss4M
-XX:ReservedCodeCacheSize=128m
-cp
"C:\Program Files (x86)\sbt\bin\sbt-launch.jar"
xsbt.boot.Boot
-Djavax.net.ssl.trustStoreType=WINDOWS-ROOT
-Dsbt.boot.credentials=C:\WORKBENCH\BPF\my-project\credentials.txt
-Dsbt.boot.directory=C:\WORKBENCH\BPF\.sbt\boot
-Dsbt.coursier.home=C:\WORKBENCH\BPF\coursier
java.lang.ClassCastException: java.lang.NoClassDefFoundError cannot be cast to xsbti.FullReload
at sbt.internal.XMainConfiguration.run(XMainConfiguration.java:59)
at sbt.xMain.run(Main.scala:46)
at xsbt.boot.Launch$.$anonfun$run$1(Launch.scala:149)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:176)
at xsbt.boot.Launch$.run(Launch.scala:149)
at xsbt.boot.Launch$.$anonfun$apply$1(Launch.scala:44)
at xsbt.boot.Launch$.launch(Launch.scala:159)
at xsbt.boot.Launch$.apply(Launch.scala:44)
at xsbt.boot.Launch$.apply(Launch.scala:21)
at xsbt.boot.Boot$.runImpl(Boot.scala:78)
at xsbt.boot.Boot$.run(Boot.scala:73)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
[error] [launcher] error during sbt launcher: java.lang.ClassCastException: java.lang.NoClassDefFoundError cannot be cast to xsbti.FullReload
Tried with different SBT versions (1.6.2, 1.5.8, 1.4.9) by changing project/build.properties): The Stack Trace differs depending on the version, but it is always a NoClassDefFoundError.
The SBT version specified in the project is successfully downloaded by the launcher.
I suspect local file permission problems as cause (e.g. due to security policies) therefore i moved the boot and the coursier cache directory. However, this did not bring the desired success.
Does anyone have an idea what the Problem could be?
(I did not do much findings about it.) In my case i could fix it by changing to appropriate java version. Initially i ran with java 18v mistakenly and then changing to java 11v it worked.
I don't have a real diagnosis or causal explanation, but I did solve a similar problem (identical error message, Linux) by clearing out the sbt cache (the .sbt folder within your user directory).
I'm trying to run a jar which accesses a save file from a predefined directory. It runs properly when running with IntelliJ, but When I run it with "java -jar", I get a NullPointerException pointing to the third line in the following snippet:
val dir = new File("./statFiles")
dir.listFiles
.filter(_.getName.takeRight(3) == "cmf")
.map(_.getName)
I get the same exception from IntelliJ if I remove the directory, so it's probably caused by the directory not being found. I'm assuming IntelliJ does something automatically that I should do manually, but I can't tell what the issue is.
edit: I've tried running the jar both from a folder that also houses the 'statFiles' folder as well as from the path in which IntelliJ creates the jar ({project root}/classes/artifacts/{name}_jar).
Stacktrace:
Exception in thread "main" java.lang.ExceptionInInitializerError
at cmsoft.ui.GUI.main(GUI.scala)
Caused by: java.lang.NullPointerException
at scala.collection.ArrayOps$.filter$extension(ArrayOps.scala:191)
at cmsoft.io.FileHandler$.findSavedStats(FileHandler.scala:24)
at cmsoft.io.FileHandler$.findSavedStat(FileHandler.scala:32)
at cmsoft.Control$.startup(Control.scala:30)
at cmsoft.ui.GUI$.<clinit>(GUI.scala:340)
... 1 more
ErrorOccuredDuringInitializationofbootlayer I keep getting this error when I run
my test:
Error occurred during initialization of boot layer
java.lang.module.FindException: Unable to derive module descriptor for
C:\Users\Bonfire.eclipse\org.eclipse.platform_4.12.0_867647348_win32_win32_x86_64\plugins\com.beust.jcommander_1.72.0.jar
Caused by: java.lang.IllegalArgumentException:
com.beust.jcommander.1.72.0: Invalid module name: '1' is not a Java
identifier
How can I fix it?
Add TestNG Library to classpath in buildpath ( not to module path). It will work fine.
To avoid this error, do not add TestNG library in the project or src folder. Try adding it into the package. To do so , follow the below mentioned procedure.
Right click on the package, click Build Path->Configure Build Path, go to Libraries tab, select Classpath and click on Add Library to select TestNG.
This error message...
Error occurred during initialization of boot layer java.lang.module.FindException:
Unable to derive module descriptor for C:\Users\Bonfire.eclipse\org.eclipse.platform_4.12.0_867647348_win32_win32_x86_64\plugins\com.beust.jcommander_1.72.0.jar
Caused by: java.lang.IllegalArgumentException: com.beust.jcommander.1.72.0: Invalid module name: '1' is not a Java identifier
...implies that there was an issue with the java classpath.
As per the discussion Launch with java 11 fails: Error: Could not find or load main class with Caused by: java.lang.ClassNotFoundException: with there are some issues with some particular classpaths for java-11 and the launch fails if:
The classpath contains a folder with spaces
The classpath length is larger than 32767 characters.
Where as java-8 works fine.
Reference
Unable to import org.openqa.selenium.WebDriver using Selenium and Java 11
Outro
Eclipse plugin: TestNG testcase does not run with Jdk11
Need to add 1.8 version library
Remove TestNG from build path library and add again
Recheck - the issue should be resolved.
I got same issue and I solved it. When creating the project, don't create module-info.java. When the project was already created with the module-info.java, deleting this file didn't help me.
I just created new project without module-info.java, and all works good now.
Best option to resolve this:
Find all file and folder starting with . e.g. — .metadata / .setting and delete them.
Add all the JARS.
Add testNG Libraries.
Execute it.
Done.
I am trying to run a Scala object called NorbertClusterClientMain which is in package com.linkedin.norbert.cluster. The source code for it is in folder examples/src/main/scala of rhavyn's open-source branch of LinkedIn Norbert, and I am working on a Linux command line.
Although I've been told that running Scala code in a package is like running Java in a package, I am in examples/src/main/scala but cannot use this command:
$ scala com.linkedin.norbert.cluster.NorbertClusterClientMain
I am getting "No such file or class on classpath", even though the file exists.
I was successfully able to compile Norbert with
$ mvn clean -DskipTests install
How can I run the NorbertClusterClientMain? Please let me know. I appreciate your help.
It is the same. So, in this case, it is looking for this file:
./com/linkedin/norbert/cluster/NorbertClusterClientMain.class
This is how Java works, and since "running" a Scala program is just running java passing the Scala library in the classpath, it has to be the same.
How did you compile it, by the way? Nevermind, saw your comment. At the directory you ran mvn, you should probably be able to run it like this:
scala -cp target com.linkedin.norbert.cluster.NorbertClusterClientMain
Failing that, find the class file, and pass the directory where com/ is to the classpath.
Your mvn script produced JAR and class files within target directory:
./target/com/.../<someClassName1>.class
./target/com/.../<someClassName2>.class
... etc
./target/<someJarName1>.jar
./target/<someJarName2>.jar
... etc
Great! Now do the same thing that you must do for java; include in your classpath:
the target base directory (this "picks up" all class files in directory hierarchy beneath target)
each jar file (this "picks up" all class files in directory hierarchy within each JAR)
scala -cp target:target/<someJarName1>.jar:target/<someJarName2>.jar:... etc ./com.linkedin.norbert.cluster.NorbertClusterClientMain
Here -cp (or equivalently, CLASSPATH environment variable) is the java classpath and so has the same syntax and rules as java.
BTW: "sbt" is a standard, powerful, usable way to build scala projects. It uses Ivy to "pull" dependencies from code repositories (i.e. mvn++). The best way to get started with it is to download sbt example projects, search for "sbt tutorial" blogs and read the sbt docs. :)
I don't anything about the code base, but that class is in the examples subproject.
This shows that it loads normally. (I haven't configured anything, because I don't know anything about the code base.)
apm#mara:~/clones/norbert$ cd examples
/home/apm/clones/norbert/examples
apm#mara:~/clones/norbert/examples$ ls
pom.xml src target
apm#mara:~/clones/norbert/examples$ mvn exec:java -Dexec.mainClass=com.linkedin.norbert.cluster.NorbertClusterClientMain
[INFO] Scanning for projects...
<snip...>
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) # norbert-examples ---
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at com.linkedin.norbert.cluster.NorbertClusterClientMain$.main(NorbertClusterClientMain.scala:22)
at com.linkedin.norbert.cluster.NorbertClusterClientMain.main(NorbertClusterClientMain.scala)
... 6 more
Here's how to add args:
https://stackoverflow.com/a/9846103/1296806
I don't use maven much anymore.
Edit: I don't use Scala 2.7.7 much anymore either.
try $ scala ./com.linkedin.norbert.cluster.NorbertClusterClientMain
or -cp .
you "current directory" might not be in classpath
Norbert uses a Scala 2.7 so directly using Scala from CLI may not work for you. Therefore find all the dependency jar using Maven and use it.
This is how I did it.
First, check out the code:
$ git clone https://github.com/rhavyn/norbert
$ cd norbert/
Build and install the dependencies in local repository first:
$ mvn clean install
Setup a classpath variable which we will use later for examples/ folder:
$ cd examples/
$ export CP=$(mvn dependency:build-classpath | grep -A1 'Dependencies classpath:' | tail -1)
Run server from examples/ folder:
$ java -cp $CP:target/classes com.linkedin.norbert.network.javaapi.NorbertJavaNetworkServerMain arg0 arg1
log4j:ERROR Could not find value for key log4j.appender.R
log4j:ERROR Could not instantiate appender named "R".
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2
at com.linkedin.norbert.network.javaapi.NorbertJavaNetworkServerMain.main(NorbertJavaNetworkServerMain.java:33)
Run client from examples/ folder:
$ java -cp $CP:target/classes com.linkedin.norbert.cluster.NorbertClusterClientMain localhost 1011
log4j:ERROR Could not find value for key log4j.appender.R
log4j:ERROR Could not instantiate appender named "R".
> h2013-12-20 13:59:44,323 - WARN [pool-1-thread-2-SendThread(0.0.3.243:2181):ClientCnxn$SendThread#1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1009)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1036)
2013-12-20 13:59:46,192 - WARN [pool-1-thread-2-SendThread(0.0.3.243:2181):ClientCnxn$SendThread#1120] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1009)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1036)
Thats it. Now all you have to do is setup the required services like Zookeper etc.